Subject: RE: [nfsv4] Sessions: Need reserved slotid values

From: [email-masked]
Date: Tue, 20 Jun 2006 17:20:37 -0400

> I guess I'm missing something.  How can the client wait
> out conflicts.  When does it know to stop waiting?

The client knows what LAYOUTGETs it's waiting for.  If the
reply to one of these is in-flight, it'll show up shortly.
If the conflicting LAYOUTGET arrives at the server after the
callback is issued, the server is REQUIRED to reject the
LAYOUTGET with an error saying that a conflict occurred (the
language in the current 4.1 draft isn't correct on this point
- I'll provide a correction as part of dealing with issue 19),
and hence the reply will be along soon.  Beyond this, I would
think that the usual operation timeout rules would apply.

> > Does Dave really want to build the entire list of conflicting
> > operations for the RECALL_FSID case? 
> 
> I guess I don't see why this such a horror.  For each slot
> the server maintains information about whether the last request
> did a layout get and what the layout it gave out was.  When
> you recall a layout or set of layouts, you go through that table
> and for each slotid that applies, (matches the fsid or matched the
> file and overlaps the range being recalled) you put in an entry 
> in the CB_SEQUENCE for the current sequence for that slot.  That 
> doesn't sound horrible and what exactly is the alternative?

SLOTID_UNKNOWN makes it the client's responsibility to clean up
whatever the client was doing.  For an entire FSID recall, it
can probably be issued faster (and there may well be a bunch
of these going out at the same time), and is more robust to
client and server getting out of sync (which shouldn't happen
in the first place, but ...).  I'm not strongly attached to this,
but it does seem useful.

Thanks,
--David
----------------------------------------------------
David L. Black, Senior Technologist
EMC Corporation, 176 South St., Hopkinton, MA  01748
+1 (508) 293-7953             FAX: +1 (508) 293-7786
[email-masked]        Mobile: +1 (978) 394-7754
----------------------------------------------------

> -----Original Message-----
> From: [email-masked] [mailto:[email-masked]] 
> Sent: Tuesday, June 20, 2006 4:00 PM
> To: Talpey, Thomas
> Cc: [email-masked]
> Subject: RE: [nfsv4] Sessions: Need reserved slotid values
> 
> 
> Does Dave really want to build the entire list of conflicting
> operations for the RECALL_FSID case?  I don't have a problem
> with an array to list the operations, but I also don't have a
> good sense of what's going to happen in practice because to a
> first approximation, HighRoad/FMP uses a single slot per file.
> 
> Another interesting question: Would a SLOTID_UNKNOWN value
> also be useful to allow a server to not check for conflicting
> operations at the cost of having the client always wait out
> conflicts? It might be useful for RECALL_FSID on a busy
> pNFS filesystem (or some other situation where the server
> wants to tell the client "just do it").
> 
> Thanks,
> --David 
> 
> > -----Original Message-----
> > From: Talpey, Thomas [mailto:[email-masked]] 
> > Sent: Tuesday, June 20, 2006 3:44 PM
> > To: Black, David
> > Cc: [email-masked]; [email-masked]
> > Subject: RE: [nfsv4] Sessions: Need reserved slotid values
> > 
> > Ok, whole-file recalls. Got it.
> > 
> > How about an alternative proposal - make the referring { sequenceid,
> > slotid } pair in the CB_SEQUENCE a counted array? (this is 
> > Dave Noveck's suggestion).
> > 
> > Then, an empty array means "none", or N entries exactly name any
> > outstanding operations. The sequencing rules handle ops 
> that are still
> > in flight to the server so the client can completely 
> disambiguate any
> > races before replying, just as in the single-op case.
> > 
> > I think this would solve the "multiple" case even better?
> >  
> > Tom.
> > 
> > At 03:25 PM 6/20/2006, [email-masked] wrote:
> > >Tom,
> > >
> > >> Can you walk me through the need for "multiple"? I am not
> > >> readily seeing how a delegation or layout recall can depend
> > >> on more than one client request. Maybe I am just being dense.
> > >
> > >Layout operations specify a range, e.g.:
> > >	--> LAYOUTGET (0-1MB for READ)
> > >	--> LAYOUTGET (2MB-3MB for WRITE)
> > >	<-- CB_LAYOUTRECALL (the whole file)
> > >
> > >Thanks,
> > >--David
> > >----------------------------------------------------
> > >David L. Black, Senior Technologist
> > >EMC Corporation, 176 South St., Hopkinton, MA  01748
> > >+1 (508) 293-7953             FAX: +1 (508) 293-7786
> > >[email-masked]        Mobile: +1 (978) 394-7754
> > >----------------------------------------------------
> > >
> > >> -----Original Message-----
> > >> From: Talpey, Thomas [mailto:[email-masked]] 
> > >> Sent: Tuesday, June 20, 2006 3:05 PM
> > >> To: Black, David
> > >> Cc: [email-masked]
> > >> Subject: Re: [nfsv4] Sessions: Need reserved slotid values
> > >> 
> > >> I definitely agree on the need for a "none" indicator in the
> > >> dependent-op reference. I'll provide text.
> > >> 
> > >> Can you walk me through the need for "multiple"? I am not
> > >> readily seeing how a delegation or layout recall can depend
> > >> on more than one client request. Maybe I am just being dense.
> > >> 
> > >> Tom.
> > >> 
> > >> At 02:50 PM 6/20/2006, [email-masked] wrote:
> > >> >I'm starting to work through the issues around applying
> > >> >sessions to potential pNFS races (19) and have run into
> > >> >a couple of problems with CB_SEQUENCE:
> > >> >
> > >> >(1) The referring slotid and sequenceid are supposed to
> > >> >	be optional (cf. section 9.5.3), but the text doesn't
> > >> >	say how to determine whether they're present vs. absent.
> > >> >
> > >> >(2) For pNFS, there could be multiple conflicting operations
> > >> >	that have to be referred to, but CB_SEQUENCE can only
> > >> >	carry one slotid/sequence reference.
> > >> >
> > >> >I think this can be fixed with a couple of reserved slotid
> > >> >values, SLOTID4_NONE, and SLOTID4_MULTIPLE - which could
> > >> >be 0xffffffff and 0xfffffffe .  pNFS does need to distinguish
> > >> >MULTIPLE from NONE, as NONE is the only case in which a client
> > >> >can immediately process a recall.
> > >> >
> > >> >I'll open this up in the issue tracker as well - I think this
> > >> >is Tom's to resolve.
> > >> >
> > >> >Thanks,
> > >> >--David
> > >> >----------------------------------------------------
> > >> >David L. Black, Senior Technologist
> > >> >EMC Corporation, 176 South St., Hopkinton, MA  01748
> > >> >+1 (508) 293-7953             FAX: +1 (508) 293-7786
> > >> >[email-masked]        Mobile: +1 (978) 394-7754
> > >> >----------------------------------------------------
> > >> >
> > >> >
> > >> >_______________________________________________
> > >> >nfsv4 mailing list
> > >> >[email-masked]
> > >> >https://www1.ietf.org/mailman/listinfo/nfsv4
> > >> 
> > >> 
> > >> _______________________________________________
> > >> nfsv4 mailing list
> > >> [email-masked]
> > >> https://www1.ietf.org/mailman/listinfo/nfsv4
> > >> 
> > 
> 
> _______________________________________________
> nfsv4 mailing list
> [email-masked]
> https://www1.ietf.org/mailman/listinfo/nfsv4
> 

_______________________________________________
nfsv4 mailing list
[email-masked]
https://www1.ietf.org/mailman/listinfo/nfsv4

---

From: "Noveck, Dave" <[email-masked]>
Date: Tue, 20 Jun 2006 16:58:51 -0400

-----Original Message-----
From: [email-masked] [mailto:[email-masked]] 
Sent: Tuesday, June 20, 2006 4:00 PM
To: Talpey, Thomas
Cc: [email-masked]
Subject: RE: [nfsv4] Sessions: Need reserved slotid values

Thanks,
--David 

> outstanding operations. The sequencing rules handle ops that are still
> in flight to the server so the client can completely disambiguate any

---

From: "Noveck, Dave" <[email-masked]>
Date: Wed, 21 Jun 2006 10:09:01 -0400

I can see your argument for SLOTID_UNKNOWN but I don't see that
you need a SLOTID_NONE value.  If you have an array, a zero-length
array takes care of this case. 

-----Original Message-----
From: [email-masked] [mailto:[email-masked]] 
Sent: Wednesday, June 21, 2006 9:48 AM
To: Talpey, Thomas; [email-masked]
Subject: RE: [nfsv4] Sessions: Need reserved slotid values

Tom,

> Alternatively, the server can simply say "none" and let the client 
> sort it all out. This may involve the client attempting to ping the 
> server in
order
> to flush in-flight operations from the TCP stream. Or, it may decide 
> to simply wait for a short time, if it believes it may have operations
outstanding.

We're almost in violent agreement, but I think you've missed an
important detail - at least in the pNFS case, "none" is not the same as
"don't know":
- In the "none" case, the client has to wait for *no* potentially
	conflicting operations because the server has said that there
	aren't any.  If the client has operations that appear to
conflict,
	the server is assuring the client that those operations did not
reach
	the server prior to the callback (and the server is promising to
	reject them if it sees them during the callback).
- In the "don't know" case the client has to wait for *every*
potentially
	conflicting operation, because the server might have processed
it
	prior to the callback, but the server isn't saying what it did.
So, "none" can't be used for "don't know".  If you agree, then I would
propose that we do the following:
	- Put an array of referenced operations into CB_SEQUENCE
	- Define SLOTID_NONE for "no conflicting operations"
	- Define SLOTID_UNKNOWN for "Server didn't check"
That would enable me to write the text needed to close out issue 19.

> I am not against having a kind of faceless "more than one", but I 
> don't
see
> why it's onerous on the server to provide the actual list, given the 
> way a session service is implemented.

Ok.  I'm not opposed to the server providing the list, but I'm
interested in allowing the server an option to "let the client sort it
all out". 

> In fact, I wonder if the server couldn't just put in a complete list 
> of the session's outstanding ops, whether or not
they
> apply to the callback. The client doesn't really do anything different
with
> each one, it just wants to be sure it has seen the server's reply.

I don't like this idea.  For pNFS, callbacks can be critical-path, and
asking a busy client  to delay processing a callback until every
outstanding op finishes is not good (and risks causing a visible
performance hiccup).

> >Another interesting question: Would a SLOTID_UNKNOWN value also be 
> >useful to allow a server to not check for conflicting operations at 
> >the cost of having the client always wait out conflicts?
> 
> I don't think so, for the reasons I mention. Besides, if we provide 
> the value, don't you think most servers will simply take the easy way 
> out and use it? :-)

See above for why it's needed in addition to NONE.  The best I can offer
about server developers is that I would add a statement that if there is
exactly one potentially conflicting operation, then it SHOULD be
provided in CB_SEQUENCE instead of using SLOTID_UNKNOWN.

Thanks,
--David

> Sent: Wednesday, June 21, 2006 9:05 AM
> To: [email-masked]
> At 03:59 PM 6/20/2006, [email-masked] wrote:
> Well, the server certainly has the list handy. It's right in the
per-session
> slot table/DRC. With a small amount of checking the server can 
> determine exactly which cached operations apply.
> 
> Alternatively, the server can simply say "none" and let the client 
> sort it all out. This may involve the client attempting to ping the 
> server in
order
> to flush in-flight operations from the TCP stream. Or, it may decide 
> to simply wait for a short time, if it believes it may have operations
outstanding.
> 
> I am not against having a kind of faceless "more than one", but I 
> don't
see
> why it's onerous on the server to provide the actual list, given the 
> way a session service is implemented. In fact, I wonder if the server 
> couldn't
just
> put in a complete list of the session's outstanding ops, whether or 
> not
they
> apply to the callback. The client doesn't really do anything different
with
> each one, it just wants to be sure it has seen the server's reply.
> 
> >with an array to list the operations, but I also don't have a good 
> >sense of what's going to happen in practice because to a first 
> >approximation, HighRoad/FMP uses a single slot per file.
> >
> >Another interesting question: Would a SLOTID_UNKNOWN value also be 
> >useful to allow a server to not check for conflicting operations at 
> >the cost of having the client always wait out conflicts?
> 
> I don't think so, for the reasons I mention. Besides, if we provide 
> the value, don't you think most servers will simply take the easy way 
> out and use it? :-)
> 
> Tom.
> 
> 
> >It might be useful for RECALL_FSID on a busy pNFS filesystem (or some

> >other situation where the server wants to tell the client "just do 
> >it").
> >
> >Thanks,
> >--David
> >
> >> How about an alternative proposal - make the referring {sequenceid,

> >> slotid } pair in the CB_SEQUENCE a counted array? (this is Dave 
> >> Noveck's suggestion).
> >> 
> >> outstanding operations. The sequencing rules handle ops that are 
> >> still in flight to the server so the client can completely 
> >> disambiguate any races before replying, just as in the single-op
case.
> >> >> readily seeing how a delegation or layout recall can depend on 
> >> >> more than one client request. Maybe I am just being dense.
> >> >
> >> >David L. Black, Senior Technologist EMC Corporation, 176 South 
> >> >St., Hopkinton, MA  01748
> >> >> readily seeing how a delegation or layout recall can depend on 
> >> >> more than one client request. Maybe I am just being dense.
> >> >> 
> >> >> Tom.
> >> >> 
> >> >> >sessions to potential pNFS races (19) and have run into a 
> >> >> >couple of problems with CB_SEQUENCE:
> >> >> >
> >> >> >values, SLOTID4_NONE, and SLOTID4_MULTIPLE - which could be 
> >> >> >0xffffffff and 0xfffffffe .  pNFS does need to distinguish 
> >> >> >David L. Black, Senior Technologist EMC Corporation, 176 South 
> >> >> >St., Hopkinton, MA  01748

---

From: [email-masked]
Date: Wed, 21 Jun 2006 10:17:54 -0400

Dave,

Ok, as long as there is a way to say both "none" and "unknown",
that's fine.

Thanks,
--David 

> From: Noveck, Dave [mailto:[email-masked]] 
> Sent: Wednesday, June 21, 2006 10:09 AM
> To: Black, David; Talpey, Thomas; [email-masked]
> > server in order
> 	aren't any.  If the client has operations that appear to conflict,
reach
> - In the "don't know" case the client has to wait for *every* potentially
> 	conflicting operation, because the server might have processed it
> > I am not against having a kind of faceless "more than one", but I don't
see
> > of the session's outstanding ops, whether or not they
with
> > Well, the server certainly has the list handy. It's right in
theper-session
> > server in order
> > I am not against having a kind of faceless "more than one", but I don't
see
> > way a session service is implemented. In fact, I wonder if the server
couldn't just
> > put in a complete list of the session's outstanding ops, whether or not
they
with
case.

---

From: "Noveck, Dave" <[email-masked]>
Date: Wed, 21 Jun 2006 12:09:45 -0400

I have to agree with Tom here.  By creating the UNKNOWN option
we are forcing the client to code for the case in which the
client doesn't know this stuff.  That would be justified
if it were the case that there were situation in which it 
would be hard for the server to know this information.  That
is different the server just not wanting to bother.

Each option we create is a potential interoperability problem
in that a client has to deal with servers that do or don't 
provide this information.  

So the question is "Why does the server *need* a don't-know
option"?  Providing one has a cost in terms of complexity and
inetroperabiltiy testing complexity.

-----Original Message-----
From: Talpey, Thomas 
Sent: Wednesday, June 21, 2006 10:56 AM
To: [email-masked]
Cc: [email-masked]
Subject: RE: [nfsv4] Sessions: Need reserved slotid values

I guess it comes down, then, to whether pNFS makes a normative
requirement on the server to inform the client of exactly what it
thinks is conflicting. Maybe I'm being thick but it seems very unwise
to let the server be so vague.

I do/did agree that "none" is different from "don't know". Basically, I
think the "don't know" is effectively the situation we were trying to
avoid by putting the referring operation(s) in the callback in the first
place. So I get confused when asked to effectively leave that out.

Sorry if I repeat myself. SLOTID_UNKNOWN is, to me, begging for a
normative statement. Why not require the server to make up its mind?

(SLOTID_NONE would be indicated by an empty array, as mentioned
earlier.)

Tom.

>> Alternatively, the server can simply say "none" and let the client
sort it
>> all out. This may involve the client attempting to ping the server in
>order
to
>> simply wait for a short time, if it believes it may have operations
>outstanding.
>
important
>detail - at least in the pNFS case, "none" is not the same as "don't
conflict,
potentially
it
don't
>> why it's onerous on the server to provide the actual list, given the
way a
>> session service is implemented.
>
interested
>in allowing the server an option to "let the client sort it all out". 
>
>> In fact, I wonder if the server couldn't just
not
>> >conflicts? 
the
>> value, don't you think most servers will simply take the easy way out
>> and use it? :-)
>
>See above for why it's needed in addition to NONE.  The best I can
>offer about server developers is that I would add a statement that
>if there is exactly one potentially conflicting operation, then it
>SHOULD be provided in CB_SEQUENCE instead of using SLOTID_UNKNOWN.
>
>Thanks,
>--David
>
determine
>> exactly which cached operations apply.
>> 
sort it
>> all out. This may involve the client attempting to ping the server in
>order
to
>> simply wait for a short time, if it believes it may have operations
>outstanding.
>> 
don't
>> why it's onerous on the server to provide the actual list, given the
way a
>> session service is implemented. In fact, I wonder if the server
couldn't
not
>> >conflicts? 
the
>> value, don't you think most servers will simply take the easy way out
>> and use it? :-)
>> 
>> Tom.
>> 
>> 
>> >It might be useful for RECALL_FSID on a busy
still