Subject: [nfsv4] Ben Campbell's Discuss on draft-ietf-nfsv4-rfc3530-migration-update-07: (with DISCUSS and COMMENT)

From: "Ben Campbell" <[email-masked]>
Date: Wed, 20 Jan 2016 12:59:02 -0800

Ben Campbell has entered the following ballot position for
draft-ietf-nfsv4-rfc3530-migration-update-07: Discuss

When responding, please keep the subject line intact and reply to all
email addresses included in the To and CC lines. (Feel free to cut this
introductory paragraph, however.)

Please refer to https://www.ietf.org/iesg/statement/discuss-criteria.html
for more information about IESG DISCUSS and COMMENT positions.

The document, along with other ballot positions, can be found here:
https://datatracker.ietf.org/doc/draft-ietf-nfsv4-rfc3530-migration-update/

----------------------------------------------------------------------
DISCUSS:
----------------------------------------------------------------------

This draft recommends the use of the uniform client-ID string approach. I
admit to not being an NFS expert, but that seems to add a lot of
complexity. It seems counter to advice in 7530. Section 4.7 of this draft
points out that this may create interop problems with some servers. It
seems to increase the privacy impact of persistent and potentially user
and hardware identifying client-ID strings. There seems to be an issue of
balancing harms here. I'd like to see some text describing how the harm
avoided by the uniform approach balances out with the other issues. 

The security considerations seem incomplete. This draft makes a number of
normative changes and clarifications that are likely to introduce new
security and/or privacy considerations that are not mentioned. This is
especially true for the guidance about using uniform client-IDs.

----------------------------------------------------------------------
COMMENT:
----------------------------------------------------------------------

I support Alissa's DISCUSS comments concerning the privacy implications
of a persistent client ID.  I am also concerned that the suggested
approaches might help enable client fingerprinting. 

- 4.2 (occurs twice)
The text says clients MAY send different strings to different servers. I
think there may be privacy and tracking implications here, even if the
client-ID is constructed without IP addresses or hardware identifiers. Is
MAY strong enough? This ties in with the recommendations throughout the
document about using persistent IDs across multiple servers. Perhaps
there should be guidance to use different strings in general, except in
cases where state-sharing is likely to come into play? (e.g. across
completely different mount points?)

-4.2, paragraph starting with "As a security measure"
It might be helpful to briefly describe the security concern.
-- paragraph starting with "The difficulty is more severe..."
I think this paragraph warrants some normative guidance.

- 4.3:
How terrible is it if client state is prematurely deleted? Much of this
draft, including the guidance to use uniform IDs across servers, seem to
treat premature deletion as an ultimate harm, and do not balance that
risk against the privacy and complexity issues that it may create.

- 4.7, 2nd bullet list:
How does the client know whether a server might return NFS4ERR_MOVED?

-7.5, 2nd bullet:
"trust relationship" by itself doesn't mean much. Who trusts what to do
what?

Editorial:

- 3, paragraph starting with "Specification of requirements for the
server..."
The first sentence hard to parse. The last sentence is also confusing. I
think you mean to say that NFV4.0 non-normatively encourages the
practice? A careless reader is likely to interpret "not 'RECOMMENDED'" as
"NOT RECOMMENDED".

-3, paragraph starting with "to further complicate matters":
The paragraph uses confusing language. I _think_ you mean that servers
have assumed a particular client implementation pattern, and can’t work
with others. But it sounds like it says clients have universally adopted
a particular pattern, in which case the interop problem doesn’t make
sense.

-3, 4th paragraph from end:
Please don't use 2119 keywords to talk about how the other text uses 2119
keywords. Or at least put "RECOMMENDs" in quotes.

- 4.2, bullet entry starting with "The verifier is a client incarnation
identifier..."
The second sentence is hard to parse. (The original version was hard, and
this version is harder.)
--bullet starting with "The string MAY...":
redundant to similar guidance earlier in section.
-- paragraph starting with "Distinct servers MAY assign..."
What do you mean by "trunking"? I don't see the term defined anywhere.

- 4.6, "... cause us to RECOMMEND use of the uniform client id string"
I don't think that sentence should use the 2119 "RECOMMEND" keyword.

- 4.8: The repeated use of first person pronouns in this section (and
elsewhere) is confusing. It's not always clear if "we" means the draft
authors, implementors, or the implementation itself.
-- paragraph before 2nd bullet list:
First sentence is hard to parse.

- 7.4.5, first paragraph:
What does it mean to "make the following notations"?

- 7.5, third bullet:
First sentence is hard to parse.

- section 7.6 and 8:
The structure is confusing. 7.6 says it modifies the security
considerations, but is not clear whether that means the security
considerations in 7530, or in this document. 8 says "It is to be modified
in section 7.6", without a clear antecedent for "It".

---

From: David Noveck <[email-masked]>
Date: Mon, 25 Jan 2016 09:11:42 -0500

> complexity.

It does add some complexity to do trunking determination, but the decision
to do so was based on the need to give migration implementations
the opportunity to work correctly.

> It seems counter to advice in 7530.

It is counter to advice in 7530 and intentionally so.   It makes the client
identity model similar to that in NFSv4.1, which uses the uniform client
id model.

> Section 4.7 of this draft
> and hardware identifying client-ID strings.

I'll discuss the general issue of privacy concerns in NFSv4 in a bit but a
few points
about the specific id's need to be made:

   - The persistence of these id's is an integral part of the design of the
   NFSv4 protocol and has been since RFC3530.
   - As mentioned in my response to Alissa, there will be changes in -08 to
   reduce/eliminate the potential of client id strings to
   be hardware-identifying.
   - With regard to the issue of identifying users, client id strings are
   not really an issue since file serving prtocols require the user to
   identify himself (and generally to provide credentials to demonstrate his
   identity)

> There seems to be an issue of

I've never thought of this as a case of balancing harms.  As I discussed
the problems seen with transparent state migration with the people who
became my co-authors and others, the issue was that the protocol feature
was not working and so the choice we had was whether to fix the problem in
v4.0, or simply give up on transparent state migration in v4.0 and wait for
v4.1 where the use of the uniform client ids meant that the basic state
management problem would not arise.

So our approach was not to treat the state management inadequacies as a
harm to be mitigated or balanced against other potential harms, but as a
 case in which the protocol did not do what it was intended to do.

That is the basis on which we proceeded in investigating the issue and
writing the document and the working group took that same approach when
deciding that this effort would continue as a working group  document..

Of course, in doing that, we tried not break anything else, but we felt
that we were not doing so for two reasons:

   - We were simply enabling use of the uniform clientid model in v4.0 that
   was already in use in v4.1.
   - In the context of a file access protocol, our privacy-related concerns
   were limited to securing content in transit from exposure to prying eyes.
   We had no concerns about other sorts of privacy and addressing this issue
   has never been considered to be an NFSv4 requirement.

> especially true for the guidance about using uniform client-IDs.

Since these changes simply adopt/recommend use of the same uniform
client id model used in v4.1, I think we should stay, with regard to
security, where we are, in line with RFC5661, until someone identifies a
concrete issue.

With regard to privacy, we need to clarify whether the privacy concerns are
 directed
to privacy against  external actors observing over-the-wire traffic or
against the server
itself.

With regard to the former, all NFSv4 specs (except RFC3010 perhaps)  have
made Kerberos privacy MANDATORY to implement and OPTIONAL to use. I don't
see any reason to change that.

WIth regard to the latter, I've already dealt with
the identification of specific users., which is necessary for the server
to provide privacy/security for the data stored.  Identification of
client nodes raises different issues in that it is not, strictly speaking,
necessary for the server to do its job.  Nevertheless typical server
implementations expect to have access to the client's IP address and use
this to supplement access control and to provide performance monitoring and
control (e.g. QOS).  Typically, NFS clients do not have
an expectation of privacy in this regard.  If that were to change,
it would require significant  work to be undertaken by the nfsv4 working
group, or development of a separate file access protocol to address the
need.

> MAY strong enough?

I think It is.

> This ties in with the recommendations throughout the

In the presence of migration, two mount points that are on different
servers at one point, can be on the same server subsequently.

> - 4.2, paragraph starting with "As a security measure"

The -08 will say "In order to prevent the possibility of malicious
destruction of the locking state associated with a client"

> I think this paragraph warrants some normative guidance.

Good point.  In -08, the id construction details section addresses this by
saying :

Note that among the items suggested for inclusion, there are many that may
conceivably change.  In order for the client id string to remain valid in
such circumstances, the client SHOULD either:

   - Use a saved copy of such value, rather than the changeable value
      itself.
      - Save the constructed client id string, rather than constructing it
      anew at SETCLIENTID time, based on unchangeable parameters and
saved copies
      of changeable data items.

The paragraph you mentioned will explicitly reference that section.

> - 4.3:
> How terrible is it if client state is prematurely deleted?

That depends on who is doing the judging.  In the case of locking state,
when a client opens a file creating a share reservation, he is being given
a promise that the file will not be accessed in particular ways by others
until the file is closed.  Loss of locking state makes it impossible to
live up to that promise.

> Much of this

"Ultimate harm" is overstating it, but it does treat the promises servers
give to clients as of a high level of importance.

> How does the client know whether a server might return NFS4ERR_MOVED?

It really can't, so -08 says:

Since the client cannot easily determine which of the above are true,
implementations are likely to rely on user-specified mount options to
select the appropriate approach to use, in cases in which multiple
approaches are supported.

> -7.5, 2nd bullet:
> "trust relationship" by itself doesn't mean much.

It doesn't give the level of detail one might like to see.  Still it is a
true statement which I hope some will find helpful.

> Who trusts what to do what?

The problem is that to explain that, one would have to discuss the
specifics of the protocol used for inter-server communication, which is out
of the scope of this document.

This is clarified in -08.

> The first sentence hard to parse.

The new text in -08 is "The need for the server to appropriately
merge stateids associated with a common client boot instance encounters a
difficult problem".

> The last sentence is also confusing. I
> practice?

Correct.

> A careless reader is likely to interpret "not 'RECOMMENDED'" as
> "NOT RECOMMENDED".

The -08 text now says "This practice is allowed and  endorsed by the
existing NFSv4.0 specification".

> with others.

Some servers have that problem.

>But it sounds like it says clients have universally adopted
> a particular pattern,

Up until recently, yes.

>in which case the interop problem doesn’t make
> sense.

Not sure you mean.  Some servers have a problem dealing with one of the
allowed approaches, which becomes an interoperability problem if some
clients now choose the disfavored-but-allowed approach.

In any case, these paragraphs now say:

The need for the server to appropriately merge stateids associated with a
common client boot instance encounters a difficult problem.  The issue is
that the common client practice with regard to the presentation of unique
strings specifying client identity makes it essentially impossible for the
client to determine whether or not two stateids, originally generated on
different servers are referable to the same client.  This practice is
allowed and  endorsed by the existing NFSv4.0 specification (RFC7530).

However, upon prototyping of clients implementing an alternative approach,
it has been found that there exist servers which do not work well with
these new clients.  It appears that current circumstances, in which a
particular client implementation pattern had been adopted universally, has
resulted in servers not being able to interoperate against alternate client
implementation patterns.  The resulting situation  requires careful
attention to compatibility issues to untangle.

> keywords. Or at least put "RECOMMENDs" in quotes.

Actually this paragraph didn't do that.   It was describing/summarizing its
intention to "RECOMMEND" certain things.

In any case, now says "recommends".

> The second sentence is hard to parse. (The original version was hard, and

The newest version is:

The verifier field contains a client boot instance identifier that is used
by the server to detect client reboots.  Only if the boot instance  is
different from that which the server had previously recorded in connection
with the client (as identified by the id field) does the server cancel the
client's leased state. This cancellation occurs once it receives
confirmation of the new nfs_clientd4 via SETCLIENTID_CONFIRM

> redundant to similar guidance earlier in section.

The earlier guidance has now been removed.

> What do you mean by "trunking"? I don't see the term defined anywhere.

The -08 has a terminology section which now says:

Trunking:

A situation in which multiple physical addresses are connected to the  same
logical server.

> I don't think that sentence should use the 2119 "RECOMMEND" keyword.

In -08, it has been changed to "recommend".

> authors, implementors, or the implementation itself.

In -08, I went through and deleted almost all uses of "we".  I think the
text is clearer even though I wound up using the passive voice more than I
like.

In -08, the only "we" left is in the Acknowledgements section where the
clear referent  is "The editor and authors of this document".

Now says:

In order to make this approach work, the client must have  certain
information accessible for each nfs_client_id4 used by the uniform approach
(only one in general).  The client needs to maintain a list of all server
IP addresses, together with the associated clientid4 values, SETCLIENTID
principals and authentication lavors.

> What does it mean to "make the following notations"?

Pretty unclear :-( That same text appeared in RFCs 3530 and 7530 as well.

I suppose the following text now in the -08 reflects the original intent:

To specify the implementation of SETCLIENTID, the following notations are
used.

> First sentence is hard to parse.

Now says:

The source server may communicate to the destination server
security-related information in order to allow it to more rigorously valid
identity

> considerations in 7530, or in this document.

Now it says "Section 19 of RFC7530".

> 8 says "It is to be modified

New version:

The security considerations of RFC7530 remain appropriate with the
exception of the modification to the penultimate  paragraph specified in
Section 8.6  of this document and the addition of the material in Section
8.5.

---

From: "Ben Campbell" <[email-masked]>
Date: Tue, 26 Jan 2016 21:33:12 -0600

Thanks for the response.

Of my two discuss points, I think the second one is likely to be handled 
as part of Alissa's similar discuss point. My intent on the first point 
is to make sure discussion happened, and it is happening. Therefore I 
plan to clear the discuss and demote the points to "no objection" 
comments. But I'd like to continue the discussion--please see inline. (I 
will remove sections that don't seem to need further discussion.)

Ben.

> identity model similar to that in NFSv4.1, which uses the uniform 

This is a point of confusion on my part. Does 4.1 assume persistent 
client ID strings across servers? I was under the impression that 4.1 
directly authenticated the client identity. (Again, I admit to being 
inexpert on NFS in general.)

> NFSv4 protocol and has been since RFC3530.

To be clear--I'm concerned about cross-server persistence, not 
persistence with the same server. Has cross-server persistence
been around since then?

> reduce/eliminate the potential of client id strings to

That will likely help.

> case in which the protocol did not do what it was intended to do.

The protocol not working seems like a harm to me :-)

I'm not saying that the protocol shouldn't be fixed. I'm saying that I 
think the draft should point out any privacy related drawbacks of the 
solution.

Would it ever be reasonable for someone to decide to forgo transparent 
state migration for privacy reasons? (I probably just climbed way out on 
the "doesn't really understand NFS" limb ...)

> has never been considered to be an NFSv4 requirement.

Well, considering keeping things private from servers:

With this guidance in this draft, would there ever be a situation where 
a user connects to server A and server B as different user identities, 
but the client implementation uses the same (or correlatable) client id 
string for both servers? Would there ever be a situation were a user 
connected to server A, then later connected to the same server with a 
different user identity, but have the same string used both times?

If a user connects to a server, disconnects, and later connects from a 
different client (but with the same user identity), can the server tell 
from the client id string that it's a different client?

If a user connects to two servers where it would never make sense to 
migrate state between them, should it still use uniform client id 
string?

> security, where we are, in line with RFC5661, until someone identifies 

At the risk of repeating a previous question: Does 4.1 involve 
persistent client id strings across different servers?

> to privacy against  external actors observing over-the-wire traffic or

I was more concerned about, for example, the ability for the server to 
infer that two different users were likely to be the same human. Or to 
infer what software implementation a given user was using. Or that a 
given NFS user and a user of some other protocol (say, http) were the 
same human.
> group, or development of a separate file access protocol to address 

[... deleted some minor discussion points that are completely dependent 
on the above discussion ...]

>> It might be helpful to briefly describe the security concern.

[...]

> This is clarified in -08.

I look forward to the clarification :-)

[...]

---

From: David Noveck <[email-masked]>
Date: Wed, 27 Jan 2016 12:08:09 -0500

> This is a point of confusion on my part. Does 4.1 assume persistent
client ID strings across servers?

Yes.  See section 2.4 of RFC5661 for the discussion.

Note that there is a difference in terminology and that RFC5661 does not
use the term "client id string".

However the co_ownerid field of the client_owner4 structure has the same
basic function and the spec says
"The string should be the same for each server network address that the
client accesses"

Section 2.4.1 discusses upgrade issue between 4.0 and 4.1 given that the
non-uniform approach
was recommended for v4.0.

> I was under the impression that 4.1 directly authenticated the client
identity. (Again, I admit to being inexpert on NFS in general.)

v4.1 contains optional feature features that allow the client identity to
be verified (SP4_MACHCRED,  SP4_SSV) if the server implements those
features and the client uses them.  Implementation of these has been
limited up until now but the primary focus of these features is preventing
hostile attacks on the NFSv4.1 replay cache.  See Section 2.10.8.3 of
RFC5661.

With regard to their potential effect client identity privacy:

   - Both of these are intended to be used in addition to the the
   co_ownerid field of the client_owner4 structure, rather than replacing it.
   - SP4_MACH_CRED, would, if anything, exacerbate the sort of privacy
   concerns you are worried about.  The client node would have to prove who he
   is and not be able easily change id's.  I'm not sure how this feature
   interacts with user-level clients but I suppose that a likely effect is to
   make it harder to obscure client node identity by using different
   user-level clients.
   - SP4_SSV, which provides mutual authentication via a shared secret,
   allows the client to prove to the server that he is not an impostor without
   exposing his actual identity to the server.   This leaves the privacy
   issues i think you are concerned about as they were with SP4_NONE, rather
   than exacerbating them.

I don't know of any infrastructure to establish such shared secrets but I
expect that all would allow there to be a way for the server to have some
sort of path back from the which particular shared secret used to some form
of correlatable identity that servers could share with one another, if they
were so inclined.  The details would depend on how this was all
administered.

> been around since then?

I misunderstood.  Cross-server persistence has been around since RFC5661.

Would it help if the following were added to the section named "Uniform
Client Id String Approach", just before the final short paragraph?

Another set of relevant considerations relate to privacy concerns, which
users of the client might have in that use of the uniform client id string
approach would enable multiple servers acting in concert to determine when
multiple requests received at different times, derive from the same NFSv4
client.  For example, this might enable determination that multiple
distinct user identities, in fact are likely to correspond to requests made
by the same person, even when those request are directed to different
servers.

> Would it ever be reasonable for someone to decide to forgo transparent
state migration for privacy reasons?

I guess I feel that, if so, that same person, if reasonable, would probably
have concerns with using NFSv4 in general.  My general impression is that
the kinds of privacy issues you are concerned with were never really
considered in the design of NFSv4.

> (I probably just climbed way out on the "doesn't really understand NFS"
limb ...)

I think you understand the protocol, but don't know about the typical
concerns and expectations of users and implementers.   Since those can
change, it is possible that some of the issues will come to be future
concerns, and, if that happens, my understanding will be lacking, and I'll
have to rethink some issues.

> With this guidance in this draft, would there ever be a situation where a
user connects to server A and server B as different user identities, but
the client implementation uses the same (or correlatable) client id string
for both servers?

Yes (same) with regard to this document and RFC5661.

Yes (correlatable) with regard to RFCs 3530 and 7530.

> Would there ever be a situation were a user connected to server A, then
later connected to the same server with a different user identity, but have
the same string used both times?

Yes.  That would be possible following the guidance in any of the RFCs
3530, 7530, 5661 and this document as well

He would easily be able do this following this document and RFC5661.

It would be harder following the instruction in RFCs 3530 and 7530, but
still possible since the assumption there is that the server would simply
append the server address to an already formed client id string prefix.

> If a user connects to two servers where it would never make sense to
migrate state between them, should it still use uniform client id string?

If the client knew (somehow) that this were the case, then the only reason
to use the same client id string would be the possible performance benefits
of trunking.

Given that the protocol gives no means of discovering this, I don't think
the spec should mention this.  After all, it is simpler for the client to
simply use the
uniform client id string model, in which case doesn't have to know or care,
where a particular file system might be migrated.

> I was more concerned about, for example, the ability for the server to
infer that two different users were likely to be the same human.

Two servers working together would be able to do that if the recommendation
in this document is followed (same id string) and also according to RFC5661.

In the context of a single server, this would be possible under all the
NFSv4 versions/documents we are discussing.

> Or to infer what software implementation a given user was using.

I suppose one could use the fact that the exact format of this string is
not specified as a way to infer the implementation.   Different
implementations would use different formats allowing the specific
implementation to be inferred.  If some is concerned about that, he could
use a user-level client to obscure what kernel he is using.

In any case, kernel clients have idiosyncrasies that someone really
interested could use to infer the software used.  It's just that
users/implementers tend not to care about this.

> Or that a given NFS user and a user of some other protocol (say, http)
were the same human.

Any possibility of that can and will be dealt with by specifying the use of
a hash function in constructing the client id string, at least as far as
v4.0 is concerned.

On Tue, Jan 26, 2016 at 10:33 PM, Ben Campbell <[email-masked]> wrote:

> as part of Alissa's similar discuss point. My intent on the first point is
> to make sure discussion happened, and it is happening. Therefore I plan to
> clear the discuss and demote the points to "no objection" comments. But I'd
> like to continue the discussion--please see inline. (I will remove sections
> that don't seem to need further discussion.)
>
> Ben.
>
> This is a point of confusion on my part. Does 4.1 assume persistent client
> ID strings across servers? I was under the impression that 4.1 directly
> authenticated the client identity. (Again, I admit to being inexpert on NFS
> in general.)
>
>
> To be clear--I'm concerned about cross-server persistence, not persistence
> with the same server. Has cross-server persistence
> been around since then?
>
> With this guidance in this draft, would there ever be a situation where a
> user connects to server A and server B as different user identities, but
> the client implementation uses the same (or correlatable) client id string
> for both servers? Would there ever be a situation were a user connected to
> server A, then later connected to the same server with a different user
> identity, but have the same string used both times?
>
> migrate state between them, should it still use uniform client id string?
>
>
>
>>
> At the risk of repeating a previous question: Does 4.1 involve persistent
> client id strings across different servers?
>
>
> infer what software implementation a given user was using. Or that a given
> NFS user and a user of some other protocol (say, http) were the same human.
>
>>
> [... deleted some minor discussion points that are completely dependent on
> the above discussion ...]
>
>

---

From: "Ben Campbell" <[email-masked]>
Date: Fri, 29 Jan 2016 16:09:15 -0600

On 27 Jan 2016, at 11:08, David Noveck wrote:

[...]

> by the same person, even when those request are directed to different

I think that would help.

[...]

	concerns, and, if that happens, my understanding will be lacking, and 
I'll

You are probably correct about my knowledge of the user expectations. 
However, we (the IETF) is trying to pay more attention to privacy issues 
these days. :-)

Thanks!

Ben.