Subject: [Crisp] xpc and threads

From: Robert Martin-Legene <[email-masked]>
Date: Fri, 13 May 2005 13:50:48 +0200

Something which struck me, but is not mentioned in the draft, is that
since each block is a seperate request they can be handled in parallel
by a server even though they may be sent one after the other via the
same TCP stream.

This could be particularly interesting in cases where lookups for some
elements are faster than for others, and maybe the client is a
"super-client" for a mass of local user-interfaces i.e. a proxy
front-end or even a busy web-server. As long as the TCP connection is
not loaded with traffic e.g. 70% of the time, you can just use a
single connection. If you get past your threshold margin, you would
just setup another TCP stream.

Obviously this directly allows you, in a simple way, to implement
front-end proxies to offload the actual TCP-burden.

In order to achieve this, you would need a request-ID for the block
when the request is sent, and the same when the answer is returned.

Unfortunately this complicates the authentication issue somewhat.

-- Robert Martin-Legène, .DK

_______________________________________________
Crisp mailing list
[email-masked]
https://www1.ietf.org/mailman/listinfo/crisp

---

From: David Blacka <[email-masked]>
Date: Fri, 13 May 2005 09:45:05 -0400

> when the request is sent, and the same when the answer is returned.

If you are going to try to do all that, you could also just use  
BEEP.  BEEP is good at multiplexing multiple streams over a single  
TCP connection.  I suggest that we not try to turn XPC into BEEP.

--
David Blacka    <[email-masked]>
Sr. Engineer    Verisign Applied Research

---

From: Andrew Newton <[email-masked]>
Date: Fri, 13 May 2005 11:08:50 -0400

> Unfortunately this complicates the authentication issue somewhat.

It complicates more than that.  Clients will need to do reordering as 
well.

While I agree that some queries can be processed faster than others, I 
don't think it is such a big deal to stop hold up the processing on the 
server while a response block is being sent.  For servers using 
blocking I/O, this is easily done by keeping the processing/writing of 
the response in the same thread as the processing/reading of the 
request (meaning a thread per connection, but that's probable happening 
anyway).  For server using non-blocking I/O, it is easy to add a 
'processing-response' flag to the TCP connection's meta-data so that 
nothing is read from the socket until the request is sent.  I find 
these both easier than having to account for multiple inbound/outbound 
blocks that could be out of sequence.

To echo Dave's response, if such a feature is needed, we ought to stick 
with BEEP.

-andy

---

From: Robert Martin-Legene <[email-masked]>
Date: Wed, 18 May 2005 14:11:41 +0200

> To echo Dave's response, if such a feature is needed, we ought to stick

I was not aware that BEEP did that, and I also like the (somewhat)
simplicity of XPC as it is now, so forget my desire to multiplex the
input and output.

I must admit though, that I still don't like a binary header. It's TCP
and we don't have the 512 byte limit. Is it really crucial, or is it
just so it looks like LWZ?

-- Robert Martin-Legène

---

From: Andrew Newton <[email-masked]>
Date: Wed, 18 May 2005 08:45:45 -0400

> just so it looks like LWZ?

I wouldn't classify it as crucial.  The motivation for binary was to 
give a similar feel to LWZ, where it is crucial.  But for me, the XPC 
binary format is very, very simple and easy to implement.  I'll admit 
that if we start getting into arbitrary name/value pairs, nested 
structures, etc..., then this binary format will not work and I'd 
rather go to a text based format (instead of falling back to something 
like ASN.1).  But what we have now is very simple and easy to do in 
binary and I do not envision a need for more complexity.

-andy

---

From: William Leibzon <[email-masked]>
Date: Wed, 18 May 2005 06:34:35 -0700 (PDT)

On Wed, 18 May 2005, Andrew Newton wrote:

> I wouldn't classify it as crucial.

> The motivation for binary was to give a similar feel to LWZ, where it 
> is crucial.

What do you mean by "similar feel"? The computer program "does not feel"
and would not care unless formats are identical and human can't read 
binary format.

> But for me, the XPC binary format 
> is very, very simple and easy to implement.  I'll admit that if we start 
> getting into arbitrary name/value pairs, nested structures, etc..., then this 
> binary format will not work and I'd rather go to a text based format (instead 
> of falling back to something like ASN.1).  But what we have now is very 
> simple and easy to do in binary and I do not envision a need for more 
> complexity.

Yes, but binary format is harder for debugging, requires special usually
numeric registries for extentions and in this case it does not seem to 
save that much data (compared to the actual data of whois database), so is 
it really worth it?

I think its serious topic to be discussed and decided before this goes
to become a standard for implementation.

---
William Leibzon
   mailto: [email-masked]
Anti-Spam and Email Security Research Worksite:
   http://www.elan.net/~william/emailsecurity/
Whois & DNS Network Investigation Tools:
   http://www.completewhois.com

---

From: Andrew Newton <[email-masked]>
Date: Wed, 18 May 2005 09:37:17 -0400

> and would not care unless formats are identical and human can't read 

Last I checked, humans program computers and humans have feelings.  Do 
you disagree?

> Yes, but binary format is harder for debugging

It's 4 bytes for a response, 5 for a request.  It's easy to eyeball 
with tcpdump.  I can't see how this is that hard.

I have implemented both a client and a server for the first version of 
this draft, and I can honestly say that is not that big of a deal.

> , requires special usually
> numeric registries for extentions

There are no new registries created here.  But I don't understand this 
point.  The IANA maintains plenty of registries for text-based 
identifiers.  An identifier is an identifier, regardless of it being 
text vs. binary, and the need for a well-known registry exists in both 
cases.

>  and in this case it does not seem to save that much data (compared to 
> the actual data of whois database), so is it really worth it?

The binary format also makes the distinction between the text data 
being transferred and the protocol bits easy to understand.  And there 
is a temptation with text formats to make extensibility easy... which 
almost always adds a lot of complexity for simple extensions.

> to become a standard for implementation.

If we want a text based protocol that can be extended and debugged and 
all the other reasons people give for text based protocols, we can just 
stick with BEEP... it is all that.

-andy

---

From: William Leibzon <[email-masked]>
Date: Wed, 18 May 2005 07:40:04 -0700 (PDT)

On Wed, 18 May 2005, Andrew Newton wrote:

> Last I checked, humans program computers and humans have feelings.  Do you 
> disagree?

I assume its rhetorical question....

The point is that humans prefer text and don't care much for binary no
matter what it is and computers would probably prefer binary (if we could
ask them :) but they really don't care how similar formats are unless they 
are identical or part of larger framework and some library can be reused.

> It's 4 bytes for a response, 5 for a request.  It's easy to eyeball with 
> tcpdump.  I can't see how this is that hard.

Text still easier. Trust me debugger program will end up being written for
this protocol to "help" the pure soles who have to do it. But you don't 
need this extra program for testing HTTP or SMTP problems.

> I have implemented both a client and a server for the first version of this 
> draft, and I can honestly say that is not that big of a deal.

I don't believe its big deal especially for the person who write the spec :)

But personally I would not mind if 4 bytes were to be passed as HTTP, SMTP 
or SIP-like header fields as they are fairly standardized and easily 
understood by humans and computers alike and have history with IETF
as way of establishing higher-level protocol interaction.

> There are no new registries created here.  But I don't understand this point. 
> The IANA maintains plenty of registries for text-based identifiers.  An 
> identifier is an identifier, regardless of it being text vs. binary, and the 
> need for a well-known registry exists in both cases.

The need and use of numeric registries is a lot higher then text ones. 
For text at least you can somehow guess what it is, even if it is not
in the registry (i.e. consider mail header fields and they are many
of them in use that are not in IANA registry) but for numeric one you
don't do something like that.

>>  and in this case it does not seem to save that much data (compared to the 
>> actual data of whois database), so is it really worth it?
>
> The binary format also makes the distinction between the text data being 
> transferred and the protocol bits easy to understand.  And there is a 
> temptation with text formats to make extensibility easy... which almost 
> always adds a lot of complexity for simple extensions.
>
> If we want a text based protocol that can be extended and debugged and all 
> the other reasons people give for text based protocols, we can just stick 
> with BEEP... it is all that.

We don't want something as complex as BEEP (actually strike that - we do
want something as complex as beep just not for everyday use :).

Something simple, yet easily understood by humans is something that I 
think is better then pure binary for start of xml transaction.

P.S. I'm willing to rewrite what you have with XPC draft into text header 
format proposal if there is interest from others in that. I'd however 
probably not bother, if I'm the only one who thinks that would be better
then pure binary.

---
William Leibzon

---

From: Andrew Newton <[email-masked]>
Date: Wed, 18 May 2005 10:59:34 -0400

On May 18, 2005, at 10:40 AM, William Leibzon wrote:
>> Last I checked, humans program computers and humans have feelings.  
>> Do you disagree?
>
> ask them :) but they really don't care how similar formats are unless 
> they are identical or part of larger framework and some library can be 
> reused.

For doing implementations, a simple binary format is easier to program, 
in my opinion.  You have focused on a protocol feature useful only for 
people sniffing packets off the network, which is not a use case we 
should encourage.

> this protocol to "help" the pure soles who have to do it. But you 
> don't need this extra program for testing HTTP or SMTP problems.

How interesting.  HTTP, BEEP and SMTP are all text based protocols but 
specific debuggers have been written for them.  So the fact that they 
are text based hasn't stopped the need for additional programs to help 
debug them.

> But personally I would not mind if 4 bytes were to be passed as HTTP, 
> SMTP or SIP-like header fields as they are fairly standardized and 
> easily understood by humans and computers alike and have history with 
> IETF

Believe it or not, I did think about doing a text based protocol.  But 
for the problem set, I decided on a very simple binary format for one 
of the reasons you have highlighted.  Many, many people look at 
examples of HTTP, SMTP, or SIP headers and think it is easy, but get 
burned because the ABNF is far more complex than they realized.  
Interpreting an octet of flags and a few octets of length is far easier 
to do than getting into foldable whitespace, line endings, character 
encoding, escaping functions for both character delimiters and binary 
data, and many of the other things that are commonly found in text 
based protocols.

>> There are no new registries created here.  But I don't understand 
>> this point. The IANA maintains plenty of registries for text-based 
>> text vs. binary, and the need for a well-known registry exists in 
>> both cases.
>

You are speaking of a generality here and something that actually isn't 
relevant to the currently defined XPC.  But since you are on this 
point, it is the "guessing" on what text based protocols do just by 
gawking at a couple of examples that have caused no end of problems.  
Just look at how many people believe they think they understand the 
protocol components of SMTP. How many times have you seen arguments on 
the true meaning of "MAIL FROM"?  The fact that a protocol makes it 
easy for people to "guess" at what it might possibly mean is not a 
benefit.

One of my other motivations for a binary format is that it makes people 
think before adding features and extensions.  We explicitly want 
something simple here.

-andy

---

From: Rick Wesson <[email-masked]>
Date: Wed, 18 May 2005 09:31:59 -0700

Robert,

The binary header will allow implementors of the protocol the ability to 
know just how big the next chunk of bits are. It enables capabilities 
much desired in protocol implementations.

I highly recomend the binary header. As an implementor I will continue 
to advocate for such -- so please don't recomend dispensing with it.

-rick

---

From: Robert Martin-Legene <[email-masked]>
Date: Wed, 25 May 2005 22:55:40 +0200

> much desired in protocol implementations.

My point was that an ascii header could do the same. I also see Andy's
PoV though. My biggest concern about binary headers is that it's
difficult to use telnet as a client. For me it's it's easier to debug
ASCII and that's all, so I won't push it.