Subject: RE: [nfsv4] Sessions: Need reserved slotid values

From: [email-masked]
Date: Wed, 21 Jun 2006 09:48:17 -0400

Tom,

> Alternatively, the server can simply say "none" and let the client sort it
> all out. This may involve the client attempting to ping the server in
order
> to flush in-flight operations from the TCP stream. Or, it may decide to
> simply wait for a short time, if it believes it may have operations
outstanding.

We're almost in violent agreement, but I think you've missed an important
detail - at least in the pNFS case, "none" is not the same as "don't know":
- In the "none" case, the client has to wait for *no* potentially
	conflicting operations because the server has said that there
	aren't any.  If the client has operations that appear to conflict,
	the server is assuring the client that those operations did not
reach
	the server prior to the callback (and the server is promising to
	reject them if it sees them during the callback).
- In the "don't know" case the client has to wait for *every* potentially
	conflicting operation, because the server might have processed it
	prior to the callback, but the server isn't saying what it did.
So, "none" can't be used for "don't know".  If you agree, then I would
propose that we do the following:
	- Put an array of referenced operations into CB_SEQUENCE
	- Define SLOTID_NONE for "no conflicting operations"
	- Define SLOTID_UNKNOWN for "Server didn't check"
That would enable me to write the text needed to close out issue 19.

> I am not against having a kind of faceless "more than one", but I don't
see
> why it's onerous on the server to provide the actual list, given the way a
> session service is implemented.

Ok.  I'm not opposed to the server providing the list, but I'm interested
in allowing the server an option to "let the client sort it all out". 

> In fact, I wonder if the server couldn't just
> put in a complete list of the session's outstanding ops, whether or not
they
> apply to the callback. The client doesn't really do anything different
with
> each one, it just wants to be sure it has seen the server's reply.

I don't like this idea.  For pNFS, callbacks can be critical-path, and
asking a busy client  to delay processing a callback until every
outstanding op finishes is not good (and risks causing a visible
performance hiccup).

> >Another interesting question: Would a SLOTID_UNKNOWN value
> >also be useful to allow a server to not check for conflicting
> >operations at the cost of having the client always wait out
> >conflicts? 
> 
> I don't think so, for the reasons I mention. Besides, if we provide the
> value, don't you think most servers will simply take the easy way out
> and use it? :-)

See above for why it's needed in addition to NONE.  The best I can
offer about server developers is that I would add a statement that
if there is exactly one potentially conflicting operation, then it
SHOULD be provided in CB_SEQUENCE instead of using SLOTID_UNKNOWN.

Thanks,
--David

> -----Original Message-----
> From: Talpey, Thomas [mailto:[email-masked]] 
> Sent: Wednesday, June 21, 2006 9:05 AM
> To: [email-masked]
> Subject: RE: [nfsv4] Sessions: Need reserved slotid values
> 
> At 03:59 PM 6/20/2006, [email-masked] wrote:
> >Does Dave really want to build the entire list of conflicting
> >operations for the RECALL_FSID case?  I don't have a problem
> 
> Well, the server certainly has the list handy. It's right in the
per-session
> slot table/DRC. With a small amount of checking the server can determine
> exactly which cached operations apply.
> 
> Alternatively, the server can simply say "none" and let the client sort it
> all out. This may involve the client attempting to ping the server in
order
> to flush in-flight operations from the TCP stream. Or, it may decide to
> simply wait for a short time, if it believes it may have operations
outstanding.
> 
> I am not against having a kind of faceless "more than one", but I don't
see
> why it's onerous on the server to provide the actual list, given the way a
> session service is implemented. In fact, I wonder if the server couldn't
just
> put in a complete list of the session's outstanding ops, whether or not
they
> apply to the callback. The client doesn't really do anything different
with
> each one, it just wants to be sure it has seen the server's reply.
> 
> >with an array to list the operations, but I also don't have a
> >good sense of what's going to happen in practice because to a
> >first approximation, HighRoad/FMP uses a single slot per file.
> >
> >Another interesting question: Would a SLOTID_UNKNOWN value
> >also be useful to allow a server to not check for conflicting
> >operations at the cost of having the client always wait out
> >conflicts? 
> 
> I don't think so, for the reasons I mention. Besides, if we provide the
> value, don't you think most servers will simply take the easy way out
> and use it? :-)
> 
> Tom.
> 
> 
> >It might be useful for RECALL_FSID on a busy
> >pNFS filesystem (or some other situation where the server
> >wants to tell the client "just do it").
> >
> >Thanks,
> >--David 
> >
> >> -----Original Message-----
> >> From: Talpey, Thomas [mailto:[email-masked]] 
> >> Sent: Tuesday, June 20, 2006 3:44 PM
> >> To: Black, David
> >> Cc: [email-masked]; [email-masked]
> >> Subject: RE: [nfsv4] Sessions: Need reserved slotid values
> >> 
> >> Ok, whole-file recalls. Got it.
> >> 
> >> How about an alternative proposal - make the referring {sequenceid,
> >> slotid } pair in the CB_SEQUENCE a counted array? (this is 
> >> Dave Noveck's suggestion).
> >> 
> >> Then, an empty array means "none", or N entries exactly name any
> >> outstanding operations. The sequencing rules handle ops that are still
> >> in flight to the server so the client can completely disambiguate any
> >> races before replying, just as in the single-op case.
> >> 
> >> I think this would solve the "multiple" case even better?
> >>  
> >> Tom.
> >> 
> >> At 03:25 PM 6/20/2006, [email-masked] wrote:
> >> >Tom,
> >> >
> >> >> Can you walk me through the need for "multiple"? I am not
> >> >> readily seeing how a delegation or layout recall can depend
> >> >> on more than one client request. Maybe I am just being dense.
> >> >
> >> >Layout operations specify a range, e.g.:
> >> >	--> LAYOUTGET (0-1MB for READ)
> >> >	--> LAYOUTGET (2MB-3MB for WRITE)
> >> >	<-- CB_LAYOUTRECALL (the whole file)
> >> >
> >> >Thanks,
> >> >--David
> >> >----------------------------------------------------
> >> >David L. Black, Senior Technologist
> >> >EMC Corporation, 176 South St., Hopkinton, MA  01748
> >> >+1 (508) 293-7953             FAX: +1 (508) 293-7786
> >> >[email-masked]        Mobile: +1 (978) 394-7754
> >> >----------------------------------------------------
> >> >
> >> >> -----Original Message-----
> >> >> From: Talpey, Thomas [mailto:[email-masked]] 
> >> >> Sent: Tuesday, June 20, 2006 3:05 PM
> >> >> To: Black, David
> >> >> Cc: [email-masked]
> >> >> Subject: Re: [nfsv4] Sessions: Need reserved slotid values
> >> >> 
> >> >> I definitely agree on the need for a "none" indicator in the
> >> >> dependent-op reference. I'll provide text.
> >> >> 
> >> >> Can you walk me through the need for "multiple"? I am not
> >> >> readily seeing how a delegation or layout recall can depend
> >> >> on more than one client request. Maybe I am just being dense.
> >> >> 
> >> >> Tom.
> >> >> 
> >> >> At 02:50 PM 6/20/2006, [email-masked] wrote:
> >> >> >I'm starting to work through the issues around applying
> >> >> >sessions to potential pNFS races (19) and have run into
> >> >> >a couple of problems with CB_SEQUENCE:
> >> >> >
> >> >> >(1) The referring slotid and sequenceid are supposed to
> >> >> >	be optional (cf. section 9.5.3), but the text doesn't
> >> >> >	say how to determine whether they're present vs. absent.
> >> >> >
> >> >> >(2) For pNFS, there could be multiple conflicting operations
> >> >> >	that have to be referred to, but CB_SEQUENCE can only
> >> >> >	carry one slotid/sequence reference.
> >> >> >
> >> >> >I think this can be fixed with a couple of reserved slotid
> >> >> >values, SLOTID4_NONE, and SLOTID4_MULTIPLE - which could
> >> >> >be 0xffffffff and 0xfffffffe .  pNFS does need to distinguish
> >> >> >MULTIPLE from NONE, as NONE is the only case in which a client
> >> >> >can immediately process a recall.
> >> >> >
> >> >> >I'll open this up in the issue tracker as well - I think this
> >> >> >is Tom's to resolve.
> >> >> >
> >> >> >Thanks,
> >> >> >--David
> >> >> >----------------------------------------------------
> >> >> >David L. Black, Senior Technologist
> >> >> >EMC Corporation, 176 South St., Hopkinton, MA  01748
> >> >> >+1 (508) 293-7953             FAX: +1 (508) 293-7786
> >> >> >[email-masked]        Mobile: +1 (978) 394-7754
> >> >> >----------------------------------------------------
> >> >> >
> >> >> >
> >> >> >_______________________________________________
> >> >> >nfsv4 mailing list
> >> >> >[email-masked]
> >> >> >https://www1.ietf.org/mailman/listinfo/nfsv4
> >> >> 
> >> >> 
> >> >> _______________________________________________
> >> >> nfsv4 mailing list
> >> >> [email-masked]
> >> >> https://www1.ietf.org/mailman/listinfo/nfsv4
> >> >> 
> >> 
> 
> 
> _______________________________________________
> nfsv4 mailing list
> [email-masked]
> https://www1.ietf.org/mailman/listinfo/nfsv4
> 

_______________________________________________
nfsv4 mailing list
[email-masked]
https://www1.ietf.org/mailman/listinfo/nfsv4

---

From: "Talpey, Thomas" <[email-masked]>
Date: Wed, 21 Jun 2006 10:55:37 -0400

I guess it comes down, then, to whether pNFS makes a normative
requirement on the server to inform the client of exactly what it
thinks is conflicting. Maybe I'm being thick but it seems very unwise
to let the server be so vague.

I do/did agree that "none" is different from "don't know". Basically, I
think the "don't know" is effectively the situation we were trying to
avoid by putting the referring operation(s) in the callback in the first
place. So I get confused when asked to effectively leave that out.

Sorry if I repeat myself. SLOTID_UNKNOWN is, to me, begging for a
normative statement. Why not require the server to make up its mind?

(SLOTID_NONE would be indicated by an empty array, as mentioned
earlier.)

Tom.

---

From: [email-masked]
Date: Wed, 21 Jun 2006 12:25:13 -0400

Tom,

> to let the server be so vague.

I think for any case we can describe concretely in terms of outstanding
operations via email is one in which the server SHOULD NOT or MUST NOT
be vague.  I can think of two cases in which a "don't be vague"
requirement might be a problem:
- The server has gotten confused, and needs the client to take
	responsibility ("stuff" happens).
- The server is doing a large-scale (FSID) recall, and prefers to
	shift responsibility to the clients.
Neither of these are compelling "absolutely have to have it, or
else something goes wrong" cases, but they could be useful.

> (SLOTID_NONE would be indicated by an empty array, as mentioned

I'll write up the issue 19 text assuming no SLOTID_UNKNOWN, and rework
the new issue to focus on it.  This sounds like fodder for discussion
in Montreal.

Thanks,
--David

> Sent: Wednesday, June 21, 2006 10:56 AM
> To: Black, David
> Cc: [email-masked]