Subject: Re new proposals

From: "Andrew Porter" <[email-masked]>
Date: Mon, 24 Nov 1997 17:14:44 -0800

I've finally had time to think out a response to the latest proposed
enhancements. I'll deal with DATA-STREAM-FLUSH first and then go on
to SUPPRESS-BYTE-DOUBLING.

A. DATA-STREAM-FLUSH

The idea seems reasonable, but let's not get ourselves twisted in a
knot trying to undo the fact that the telnet abort output (AO)
command got (mis)used for something else. Presuming it had been
agreed upon during functions negotiation, how about implementing it
as a suboption? The sender would write

IAC SB TN3270E DATA-STREAM-FLUSH IAC SE

to tell the receiver to discard the TN3270E packet he's been
accumulating.  It may seem odd to send a suboption packet
sometime other than initial negotiation, but we can't stick
something inside a TN3270E header or data -- that's what we're
trying to cancel in the first place -- and a TN3270E suboption
buffer is the only other place that we're free to use.

B. SUPPRESS-BYTE-DOUBLING

A quote from the draft proposal:
  "One of the performance bottlenecks facing server and client
   vendors is that any data stream containing '0xFF' must be
   byte-doubled in order to differentiate actual data from
   IAC/Telnet Command."

Once upon a time I thought this was a "performance bottleneck", too.
5 years ago or so I designed and (on the client side) implemented a
solution similar to the proposal as a private extension between our
(I/O Concepts) clients and servers in what was otherwise a TN3270
data stream. My opinions based on a several years of real world
experience:

a) Speaking from the client perspective (and isn't the end user always
   a client?) we've got a protocol right now that's fairly easy to
   implement using a layered architecture. My telnet layer understands
   telnet protocol and otherwise accumulates data; when it gets an
   IAC EOR it passes the buffer to the data stream layer. The data
   stream layer understands TN3270E headers and TN3270 data. Nice,
   clean, easy to maintain. I don't like the idea of having a
   variable-length TN3270E header and having to interpret the
   TN3270E header at the telnet layer (later on I'll propose a
   different solution that takes care of this objection).
b) The current byte-oriented nature of the current protocol may seem
   wasteful but it does have the virtue of reliability. When you
   stop looking at every byte and start sending blocks of data then
   you have to reassemble them correctly -- just because the sender
   writes 1800 bytes at once doesn't mean the receiver gets them in
   one chunk, it might come in (for example) separate packets of
   1200 bytes then 600. It makes for boundary condition bugs that are
   randomly dependent on the local network and hence difficult to
   reproduce and debug. It was painful in a controlled situation
   involving one client (and one programmer) and one server (and one
   other programmer) to get this working reliably...
c) Performance gains. Once it was working the end result was not
   as good as we had hoped. The transactions that did benefit were
   those involving large structured field buffers (IND$FILE file
   transfers, 3179-G graphics data), maybe a 10% improvement in
   speed as I recall; in fact we wound up implementing the
   procotol in such a fashion that "small" (<= 128 bytes) data
   buffers continued to be sent in the byte-doubled way...
   Not exactly the elimination of a "performance bottleneck".

The end result is I'm dubious as to the usefulness of this proposal.

Having said that I also feel compelled to suggest a different way of
doing this. I think that the TN3270E header is really an intrinsic
part of the data that we're trying to "block up" and that it's not
an appropriate place for the "blocking" info. Remember the proposed
DATA-STREAM-FLUSH function? It implies a server that is sending
3270 data to a client immediately upon receipt of the RU, rather
than waiting until it gets to the last RU. If we say that the
blocking information is going to be in the TN3270E header, then
this server has a choice of a) not using SUPPRESS-BYTE-DOUBLING
or b) not sending the data immediately but instead accumulating
it the last RU, in effect rendering DATA-STREAM-FLUSH meaningless.
Actually I suppose there's another choice, too -- allow the server
to use SUPPRESS-BYTE-DOUBLING but require it to send each packet
with a TN3270E header in front -- but that seems unappetizing: now
we've got multiple TN3270E headers for what's really a single 3270
buffer?

I suggest instead a solution that is similar to my earlier proposal
for DATA-STREAM-FLUSH, namely send the blocking information in a
TN3270E suboption packet:

IAC SB TN3270E BLOCK-DATA LENGTH1 LENGTH2 IAC SE

This promises that the next "length" bytes following the SE are
non-byte-doubled 3270 (including TN3270E header) data, but says
nothing about it being complete -- we keep the IAC EOR to flag the
end-of-record. Conceivably, then, a large buffer could be sent as
follows:

IAC SB TN3270E BLOCK-DATA LENGTH1 LENGTH2 IAC SE

<"length" data, TN3270E header and part of the 3270 data >

IAC SB TN3270E BLOCK-DATA LENGTH1 LENGTH2 IAC SE

<"length" data, the rest of the 3270 data>

IAC EOR

Servers and clients could send data using either the traditional
byte-doubled or non-byte-doubled methods.

Thanks for listening!

******************************
*  Andrew Porter             *
*  Senior Software Engineer  *
*  I/O Concepts              *
*  [email-masked]       *
******************************

---

From: "Derek W. Bolton" <[email-masked]>
Date: Tue, 25 Nov 1997 14:02:16 +1100 (EST)

Andrew,

> a) Speaking from the client perspective (and isn't the end user
>    always a client?) we've got a protocol right now that's fairly
>    easy to implement using a layered architecture. My telnet layer
>    understands telnet protocol and otherwise accumulates data; when
>    it gets an IAC EOR it passes the buffer to the data stream
>    layer. The data stream layer understands TN3270E headers and
>    TN3270 data. Nice, clean, easy to maintain. I don't like the idea
>    of having a variable-length TN3270E header and having to
>    interpret the TN3270E header at the telnet layer (later on I'll
>    propose a different solution that takes care of this objection).

>    1200 bytes then 600. It makes for boundary condition bugs that
>    are randomly dependent on the local network and hence difficult
>    to reproduce and debug. It was painful in a controlled situation

I also have had experience of a length-and-data block mode over
a streamed transport.  There were no problems arising from it.

> The end result is I'm dubious as to the usefulness of this proposal.

In my view, the performance hit is much more of an issue at the
server end.  The client has a high CPU/byte cost anyway, so
the extra for the FF processing is only a small percentage.
At the server end the same per byte delta represents a much
higher percentage.  But I don't have any figures.

> byte-doubled or non-byte-doubled methods.

Apart from the fact that you're sending an 8-byte block-introducer,
whereas I had a mode-setting message and a 2-byte block-introducer,
I believe your proposal is identical to the one I sent out yesterday.
So of course, I agree with your suggestion!

Regards,
  Derek