Subject: Re: Multiple extended responses per request

From: "Steve Fisher" <[email-masked]>
Date: Mon, 13 Sep 1999 09:23:50 +0100

Dave

 In some ways, your requirements seem similar to the X.500 DAP pagedResults
feature which organizes search responses into a set of 'pages'. This is
initiated by the client in the initial search request (using a non-critical
extension) and then collected in page size chunks of entries in subsequent
searchRequests. This is fine if the client anticipates that some type of
management of responses is required, but is not something that the Server
can initiate (as far as I can see), and which is something that wasnt clear
to me in your original message. If you are looking to the Server to prompt
multiple results, if the client is getting back more than it anticipated, it
would ordinarily perform a more refined operation rather than accept a lot
of unexpected entries.

An extended operation which deals with searches (or maybe two - one to
initiate the orgainisation of results, the other to collect existing results
'pages') which would seem a reasonable thing to do as it would maintain the
single request/response model and the operation of it would be very similar
to the existing search results processing procedures.

If this is a 'general feature', (i.e. the organization of any operation into
multiple responses) then this couldn't be an extended operation, as you
potentially would like to 'multiply respond'  to any extended (or core)
operation.

Steve Fisher

BT Syncordia
[email-masked]

-----Original Message-----
From: Dave Wilbur <[email-masked]>
To: [email-masked]
<[email-masked]>; [email-masked]
<[email-masked]>
Cc: [email-masked] <[email-masked]>
Date: 10 September 1999 17:38
Subject: Re: Multiple extended responses per request

Thanks for the feedback from Bruce and Kurt.

The suggestion that each response be prompted by a
request could also be used for Persistent Search (as well
as a standard Search, for that matter).  The fact that Search
saw a need (to reduce overhead, other?) to allow for multiple
responses to a request, in the form of entries and a result
leads me to believe that an extension should be able to
accomplish similar functionality.

The fact that Kurt suggested that we CAN do multiple
responses per exteded request leads me to believe that
there are others who think this should be allowed.  Currently
the RFC implies that you only get one Extended Response
per request and the client SDKs are implemented so that you
can only retrieve one response per request.  So if we are to
allow multiple resonses per request, then the RFC should state
this and the SDKs should support it.

Dave

>>> "Kurt D. Zeilenga" <[email-masked]> 9/9/99 10:13:51 AM >>>
At 08:48 AM 9/9/99 -0700, Bruce Greenblatt wrote:
>Hi Dave!
>
>You'll want to do something similar to the multi-stage bind.

Or something similiar to what search does:

CLIENT                                               SERVER
              Extended Request
        ------------------------------------------>

              Extended Response (not done)
        <------------------------------------------
              Extended Response (not done)
        <------------------------------------------
              Extended Response (not done)
        <------------------------------------------
              Extended Response (not done)
        <------------------------------------------
              Extended Response (not done)
        <------------------------------------------
              Extended Response (not done)
        <------------------------------------------
              Extended Response (done)
        <------------------------------------------

At 03:50 PM 9/8/99 -0600, Dave Wilbur wrote:
>While developing an LDAP v3 extension, a developer here wanted to be able
to send multiple responses against one request.  This was relatively easy
to support in our server code, but the published client APIs don't support
more than one response per message id.  While RFC 2251 for LDAPResult is
quite clear in stating "to indicate the *final* status of a protocol
operation request", the ExtendedResponse is not quite so definitive: "The
server will respond to this with an LDAPMessge containing the
ExtendedResponse."
>
>Given the precedent of an interim response in the Search verb as well as
Persistent and Triggered search, it seems to make sense that an Extension
might want to do this as well.  We were going to use different OIDs to
indicate if a response was interim or final until we ran into the client
issue.
>
>Is this feasible? reasonable?
>
>Looking for feedback.
>
>Dave

---

From: steve fisher <[email-masked]>
Date: Tue, 14 Sep 1999 17:00:51 +0100

-----Original Message-----
From: David Boreham <[email-masked]>
To: [email-masked]' <[email-masked]>
Date: 14 September 1999 16:23
Subject: Re: Multiple extended responses per request

>Gil Kirkpatrick wrote:
>
>> If the purpose is to provide progress information to the client, doesn't
>> this scheme generate an inordinate amount of traffic, e.g. in your
example
>> of the tree delete, a packet per entry in the tree being deleted?
>

You could use percentage indicators.

>I wonder if this is a practical problem ?
>
>Another idea I've heard
>discussed which _could_ produce less traffic is to allow the
>client to monitor the progress of long-running operations
>like this via magic DIT entries. Somehow an entry would be
>created for the operation (e.g. long search, long subtree
>delete). That entry would have attribute values which tell
>the client how much progress has been made by the server
>in processing the operation. I'm not sure if this is a
>terribly good idea or not. It's obviously quarter-baked.

If we are to use a client-driven approach then, although this looks like a
nice simple mechanism, the function that it provides is a management one. If
this was the only type of management activity it may be worth considering,
but I doubt whether this would be the case. It may be better to use/define
managed objects instead of magic DIT entries and use the measuring
facilities of existing managementn protocols , incorporate a management
client in with the ldap client, and co-ordinate the operation of the two
clients.

---

From: "Roger Harrison" <[email-masked]>
Date: Tue, 14 Sep 1999 11:14:10 -0600

My example is hypothetical and is only to be used to illustrate one possible reason why you'd one multiple responses for an extended request.  Each extension would need to be designed so that information sent on the wire was done in an efficient and reasonable way.  The key is to give the extension designers the ability to use multiple responses so we has a choice in the first place.

Roger

>>> Gil Kirkpatrick <[email-masked]> 09/14/99 09:15AM >>>
Roger,

If the purpose is to provide progress information to the client, doesn't
this scheme generate an inordinate amount of traffic, e.g. in your example
of the tree delete, a packet per entry in the tree being deleted?

Gil

> From:	Roger Harrison [SMTP:[email-masked]] 
> Sent:	Tuesday, September 14, 1999 12:31 AM
> To:	<
> Based on my discussions with Dave Wilbur (we both work at Novell), the
> idea for multiple responses to an extended request came from the desire to
> get status on directory operations that might take a lot of time rather
> than the need to collate results in some way.  
> 
> Imagine a directory operation that could take a "long" time--deleting a
> very large subtree, for instance.  It would be nice for the server to
> provide feedback during the time the operation is completing so that the
> client is just left hanging until the operation finally completes.  In
> this imaginary example, I think the dialog might look like this:
> 
> C  -> S: Delete Subtree 
> S -> C: Deleted entry  a
> S -> C: Deleted entry  b
> S -> C: Deleted entry c
> ...
> S -> C: Deleted entry x
> S -> C: Deleted entry y
> S -> C: Deleted entry z
> S -> C: Result: Success
> 
> The client would have to know and expect multiple responses for an
> extended request of this type, just like it does for search.   I don't see
> a problem with this since the extended operation would define the allowed
> behavior, and the client could therefore be written accordingly. Since the
> client initiates the extended operation, it would know to expect the
> associated behavior as well.
> 
> Roger
> 
> ------------------------------------------------------
> Roger G. Harrison
> Novell, Inc.
> [email-masked] 
> 
> >>> "Steve Fisher" <[email-masked]> 09/13/99 03:29AM >>>
> Dave
>

---

From: "Roger Harrison" <[email-masked]>
Date: Tue, 14 Sep 1999 11:36:29 -0600

A client polling mechanism would work, but I think that there will be times when it would be too inefficient.  Imagine having to do a subtree search request followed by a myriad of polling operations to get back all of the entries.  It wouldn't be very efficient, hence we have a single-request-multiple-response model for searching.  Allowing the same response model for extended operations doesn't preclude using client polling where it makes more sense, but forcing extensions to have a single-request-single-response model does preclude extension designers from making potentially useful design decisions.

It seems that the point of extensions was to allow LDAP to grow as needed.  I'm not an advocate of an poorly planned, free-for-all protocol, but I am an advocate of technical choice.

Roger

>>> Gil Kirkpatrick <[email-masked]> 09/14/99 11:10AM >>>
I have to agree with Mark, making LDAP clients rely on an extra-LDAP
protocol like SNMP sounds like a bad idea. I would say the purpose of SNMP
is to provide management information about the state of the service or
device, not the state of an individual operation (except possibly as it
relates to service management).

A mechanism where the client polls (using LDAP) for status information would
be more appropriate for long operations. This would let the client poll when
it needed an update (if it needed it) as opposed to having to handle a
status message for every object affected by the operation. Perhaps the
operation could return an operation identifier (e.g. a handle) that the
client would use as part of the status request.

Another way to reduce the notification traffic from the service would be to
include some parameters with the control that would throttle the
notification messages to one notification every N seconds or one
notification every N objects.

-gil

> From:	Bruce Greenblatt [SMTP:[email-masked]] 
> Sent:	Tuesday, September 14, 1999 10:02 AM
> To:	[email-masked] 
> Cc:	Roger Harrison; [email-masked] 
> It wold certainly be easy enough to put together a prototype of an SNMP
> status reporting thing.  That's the way that I'd try and see if the dual
> protocol approach is workable.  Since "everyone" already supports the dsa
> MIB, it seems like a possibility that this is a natural addon to the
> dsaOpsTable.  
> 
> At 12:44 PM 9/14/99 -0400, Mark C Smith wrote:
> >I for one am skeptical of the value in using another protocol (e.g.,
> >SNMP) to retrieve status about LDAP requests.  That just doesn't feel
> >like the right architectural solution to me.  Coordinating requests and
> >responses across two different protocols on the client side will be
> >awkward.  I do think the topic of how best to provide status about "long
> >lived" LDAP operations is an important one.
> >
> >-- 
> >Mark Smith
> >iPlanet Directory Architect / Sun-Netscape Alliance
> >My words are my own, not my employer's.   Got LDAP?
> >
> >
> >Bruce Greenblatt wrote:
> >> 
> >> Roger,
> >> 
> >> It seems to me that SNMP would be more suited to providing this
> feedback.
> >> It would be a simple matter to define a mib that AUGMENTS the DSA mib
> to
> >> report this feedback.  You'd have one row in the MIB for each
> outstanding
> >> operation that you're providing feedback on.  You could have whatever
> >> columns you want to provide details about the operation, and the
> current
> >> status...
> >> 
> >> Bruce
> >> 
> >> At 01:30 AM 9/14/99 -0600, Roger Harrison wrote:
> this
> >> imaginary example, I think the dialog might look like this:
> >> >
> >> >C  -> S: Delete Subtree
> >> >S -> C: Deleted entry  a
> >> >S -> C: Deleted entry  b
> >> >S -> C: Deleted entry c
> >> >...
> >> >S -> C: Deleted entry x
> >> >S -> C: Deleted entry y
> >> >S -> C: Deleted entry z
> >> >S -> C: Result: Success
> >> >
> >> >'pages') which would seem a reasonable thing to do as it would
> maintain
> the
> >> >If this is a 'general feature', (i.e. the organization of any
> operation
> into
> >> ==============================================
> >> Bruce Greenblatt, Ph. D.
> >> Directory Tools and Application Services, Inc.
> >> http://www.directory-applications.com 
> >
> >
> >
> ==============================================
> Bruce Greenblatt, Ph. D.
> Directory Tools and Application Services, Inc.
> http://www.directory-applications.com

---

From: "Salter, Thomas A" <[email-masked]>
Date: Tue, 14 Sep 1999 15:46:36 -0400

If I've followed this correctly, there are two issues:
1. People want to extend Ldap with operations that send multi-part responses
as search does.
2. Client APIs need to treat these kinds of responses specially.

Some possible solutions are:
1. Define a control that can be used with any response and simply says "this
is not the last message".
2. Same as 1, but the control can only be used with ExtendedResponse.
3. Define a new message type called ExtendedPartialResponse.  Revise RFC2251
to say that the server responds to an ExtendedRequest with zero or more
ExtendedPartialResponse messages followed by one ExtendedResponse.
4. Same as 3, but allow ExtendedPartialResponse messages to be returned by
any Ldap request.

Interestingly, RFC2251, 4.5.2 says that an ExtendedResponse can be sent in
response to a SearchRequest, yet it doesn't specify precisely how that
should work -- is it in addition to or instead of the SearchResult.

---

From: "Roger Harrison" <[email-masked]>
Date: Tue, 14 Sep 1999 13:53:07 -0600

This is an accurate summary according to my point of view.  I would vote for defining an Extended Partial Response to solve the issue of allowing multi-part responses to be formally allowed by the protocol for extended operations.  The remaining issue of Client APIs would still need to be addressed since their current incarnation doesn't appear to handle multi-part responses to an extended request.

Roger

>>> "Salter, Thomas A" <[email-masked]> 09/14/99 01:49PM >>>

---

From: "Roger Harrison" <[email-masked]>
Date: Tue, 14 Sep 1999 13:53:07 -0600

Roger

---

From: steve fisher <[email-masked]>
Date: Wed, 15 Sep 1999 09:00:04 +0100

-----Original Message-----
From: Mark C Smith <[email-masked]>
To: Bruce Greenblatt <[email-masked]>
Cc: Roger Harrison <[email-masked]>; [email-masked]
<[email-masked]>
Date: 14 September 1999 17:45
Subject: Re: Multiple extended responses per request

>lived" LDAP operations is an important one.

Coordinating activity across two protocols isnt a problem, its the
co-ordination of two different types of activity. This will be the case
anyway, whether LDAP is augmented to carry management information or whether
snmp or some other management protocol is used.

I think that there is an issue as to how much of the server management
information is useful to the client. The original post was a hypothetical
example, so I would imagine that if there is  a requirement to give the
client access to mgmt info, there would be more than one type. If this is
the case I do not see how this would be a useful thing for LDAP to cover.
Management protocols already have the core management functions in place, to
do this again with LDAP would be difficult to justify. If any activity is to
take place in this area, I think that a  MIB for client accesible
information based on a management protocol would be the best route.

Steve Fisher

BT Syncordia

>> It seems to me that SNMP would be more suited to providing this feedback.
>> It would be a simple matter to define a mib that AUGMENTS the DSA mib to
>> report this feedback.  You'd have one row in the MIB for each outstanding
>> columns you want to provide details about the operation, and the current
>> status...
>>
>> Bruce
>>

---

From: steve fisher <[email-masked]>
Date: Wed, 15 Sep 1999 09:04:47 +0100

-----Original Message-----
From: Kurt D. Zeilenga <[email-masked]>
To: [email-masked] <[email-masked]>
Date: 14 September 1999 17:55
Subject: Re: Multiple extended responses per request

>I think there are likely many operations were it makes
>sense to return multiple responses to one request.  Search
>is one such operation.  There are likely others which make
>as much sense as search.
>
>A subtree delete is a good example.  Besides returning results
>for each entry deleted, it may need to return references and
>such.  Today, a subtree delete could be implemented using a
>search PDU with a delete control.
>
>I guess we either live with abuse of the search operation or
>we live with multiple extended responses to a extended request.

To allow multiple responses for extended operations, in a similar way to
search responses would probably mean a modification to the ExtendedResponse
message, to change the eay that the LDAPResult field is included

Steve Fisher

BT Syncordia

>Kurt

---

From: Gil Kirkpatrick <[email-masked]>
Date: Wed, 15 Sep 1999 08:55:48 -0700

If you provide for "detached processes", i.e. long-duation operations that
can continue beyond the scope of the connection that created them, it seems
like you're creating a whole new mechanism for the server. It would
effectively provide a batch procesing facility where clients submit commands
to be processed, and then disconnect from the server. The status of the
commands can then be monitored from anywhere using SNMP.

A neat idea, and not hard to put together a prototype. It also takes the
status notification issue out of the LDAP protocol, which may be a good
thing. Though I don't think this is where the conversation wants to go;
Roger just wants the option of multiple responses in extensions, if I
understand him properly.

-gil

> Sent:	Wednesday, September 15, 1999 8:46 AM
> To:	Gil Kirkpatrick; [email-masked]
> Subject:	RE: Multiple extended responses per request
> 
> There are a couple of reasons that you might want to consider using SNMP.
> If you only report status via LDAP, status can only be reported to the
> client that issued the initial request.  Also, assuming the basic, obvious
> implementation, the connection must remain alive for the life of the long
> lived operation.  I can imagine a subtree copy of several million objects
> that could take many hours to complete.  If you use SNMP, the status can
> be
> retrieved from any management station.  I guess that this is analogous to
> using the stored LDAP object approach that was also suggested... Bruce
> 
> At 10:10 AM 9/14/99 -0700, Gil Kirkpatrick wrote:
> >> From:	Bruce Greenblatt
> [SMTP:[email-masked]]
> >> >> idea for multiple responses to an extended request came from the
> desire
> >> to
> >> >> >Imagine a directory operation that could take a "long"
> time--deleting
> >> a
> >> >> extended request of this type, just like it does for search.   I
> don't
> >> see
> >> >> behavior, and the client could therefore be written accordingly.
> Since
> >> the
> >> >> >feature which organizes search responses into a set of 'pages'.
> This
> >> is
> >> >> >searchRequests. This is fine if the client anticipates that some
> type
> >> of
> >> >> >can initiate (as far as I can see), and which is something that
> wasnt
> >> clear
> >> >> >would ordinarily perform a more refined operation rather than
> accept a
> >> lot
> >> >> >of unexpected entries.
> >> >> >
> >> >> >initiate the orgainisation of results, the other to collect
> existing
> >> results
> >> >> >>While developing an LDAP v3 extension, a developer here wanted to
> be
> >> able
> >> >> >to send multiple responses against one request.  This was
> relatively
> >> easy
> >> >> >more than one response per message id.  While RFC 2251 for
> LDAPResult
> >> is
> >> >> >operation request", the ExtendedResponse is not quite so
> definitive:
> >> "The
> >> >> >>Given the precedent of an interim response in the Search verb as
> well
> >> as

---

From: "Roger Harrison" <[email-masked]>
Date: Wed, 15 Sep 1999 11:04:19 -0600

The idea of using SNMP for monitoring purposes is very appealing for situations where monitoring status is the desired task.  In fact, Bruce's idea has given me some things to consider in some other work I'm doing. Thanks!

Gil correctly points out that my (actually Dave Wilbur's) original intent here was just to have the option of multiple responses available to extensions when what they're doing warrants it.  Searching is an example of an operation that benefits greatly from this paradigm, and I imagine that other operations may similarly benefit.  If this isn't a good enough reason for changing RFC 2251 to allow multiple responses to an extended request, then I guess we'll have to come up with a practical, concrete example that is more convincing to the LDAP community.

Roger

>>> Gil Kirkpatrick <[email-masked]> 09/15/99 09:57AM >>>

-gil