Subject: Re: [Netconf] how to handle XML parse errors

From: Ladislav Lhotka <[email-masked]>
Date: Wed, 01 Sep 2010 09:02:46 +0200

On Wed, 18 Aug 2010 10:53:44 -0400, Phil Shafer <[email-masked]> wrote:
> Juergen Schoenwaelder writes:
> >Closing the session is a drastic measure but given the shortcomings of
> >the framing we have invented, this might be the most robust solution
> >overall, although likely a bit annoying on the cut'n'paste interface.
> 
> Any chance we can address the framing issue?  It would be nice to
> have something real, workable, and robust here.

By conincidence, I just found out there in fact exists a robust
end-of-frame marker, actually 29 of them: characters in the range
0x0-0x1F, except CR, LF and TAB, are not legal in XML documents at all
[XML 1.0, section 2.2]. I know it's rather late, but perhaps it's still
worth considering for 4742bis.

Lada

> 
> Thanks,
>  Phil
> _______________________________________________
> Netconf mailing list
> [email-masked]
> https://www.ietf.org/mailman/listinfo/netconf

-- 
Ladislav Lhotka, CESNET
PGP Key ID: E74E8C0C

---

From: "Randy Presuhn" <[email-masked]>
Date: Wed, 1 Sep 2010 10:56:44 -0700

Hi -

> From: "Ladislav Lhotka" <[email-masked]>
> To: "Phil Shafer" <[email-masked]>; "Juergen Schoenwaelder" <[email-masked]>
> Cc: <[email-masked]>
> Sent: Wednesday, September 01, 2010 12:02 AM
> Subject: Re: [Netconf] how to handle XML parse errors
...
> worth considering for 4742bis.

While C programmers might be tempted by 0x00, for some reason
0x07 rings a bell.  :-) 

Randy

---

From: Ladislav Lhotka <[email-masked]>
Date: Thu, 02 Sep 2010 08:15:13 +0200

> While C programmers might be tempted by 0x00, for some reason

The octal form is even more attractive. :-) But seriously, 0x04 (ASCII
"END OF TRANSMISSION") would seem a logical choice.

Lada

-- 
Ladislav Lhotka, CESNET
PGP Key ID: E74E8C0C

---

From: Martin Bjorklund <[email-masked]>
Date: Fri, 03 Sep 2010 09:36:17 +0200 (CEST)

> worth considering for 4742bis.

Note that this change would be backwards incompatible on the transport
layer, i.e., essentially a new transport.  There is no way a peer can
learn which EOM marker the other peer is using.  The solution to this
would be a new port and/or subsystem name, and I do not think it is
worth it.

/martin

---

From: Ladislav Lhotka <[email-masked]>
Date: Fri, 03 Sep 2010 10:23:39 +0200

> would be a new port and/or subsystem name, and I do not think it is

Yes, what I am proposing is to abandon the old marker because it is
broken and there is a safer alternative. This statement in 4742bis is
simply false:

  [ The ']]>]]>' ] character sequence cannot legally appear outside of
  a comment in an XML document, so it can be unambigiously used to
  identify the end of the current document, allowing resynchronization
  of the NETCONF exchange in the event of an XML syntax or parsing
  error.

The sequence may legally appear, beside comments, in processing
instructions and, most importantly, attribute values. On the practical
side, a NETCONF message that is - from the XML point of view - not only
well-formed but also valid according to a data model schema can make the
server kill the session (which is still a better case).

Lada  

-- 
Ladislav Lhotka, CESNET
PGP Key ID: E74E8C0C

---

From: Phil Shafer <[email-masked]>
Date: Fri, 3 Sep 2010 09:45:34 -0400

>would be a new port and/or subsystem name, and I do not think it is

This would break backwards compatibility, but would address a real
problem.  We could define a new subsystem for ssh, like "netconf-0x04",
that using the 0x04 byte to end messages.  This would completely
close the hole and do it in a single byte that can be easily detected.
The client can attempt to use the new subsystem, and fallback to
the old "netconf" one if it fails.  The client can "remember" which
subsystem is supported by each target, and avoid this try-and-fallback
is desired.

Thanks,
 Phil

---

From: David Reid <[email-masked]>
Date: Fri, 03 Sep 2010 10:28:01 -0400

> worth considering for 4742bis.

Non-printable characters would be a bit inconvenient for cut-n-paste.

-David Reid

---

From: Ladislav Lhotka <[email-masked]>
Date: Fri, 03 Sep 2010 17:27:15 +0200

> Non-printable characters would be a bit inconvenient for cut-n-paste.

Yes, but for those who want to type NETCONF to the console, it's easy to
write a simple filter that will add the control character.

Also, if readability was a concern, the 0x04 character could be
(optionally) preceded by a processing instruction, e.g. <?EOM?>.

Lada

-- 
Ladislav Lhotka, CESNET
PGP Key ID: E74E8C0C

---

From: Andy Bierman <[email-masked]>
Date: Fri, 03 Sep 2010 08:36:34 -0700

> would be a new port and/or subsystem name, and I do not think it is

I agree.
It would be an even bigger mess when you consider the base:1.0 vs. base:1.1
distinction.  Relying on the default port number is not an option because
the operator can configure the server to use a non-default port.

The base:1.1 URI is still needed to allow 1.1 over the other transports.
That cannot be used in this case since the NETCONF peer needs to
de-frame the <hello> message before reading it to find out if 'base:1;1'
is present or not.

Using a new SSH subsystem removes the possibility that a 1.1 client
can use a single session login attempt to try for a 1.1 or a 1.0 session.
Instead, the client will end up trying a 1.1 session on the new subsystem,
and waiting the full timeout interval before giving up,
and starting over with the old 1.0 subsystem.

It is operationally unacceptable to introduce timeouts into the protocol
version negotiation algorithm.  The current solution in 4741bis does not add any additional
message flows or session login attempts to let the client and server pick
the most recent version in common.

> /martin

Andy

---

From: Juergen Schoenwaelder <[email-masked]>
Date: Fri, 3 Sep 2010 17:57:31 +0200

> de-frame the <hello> message before reading it to find out if 'base:1;1'

Yes, you should not mix different layers.

> and starting over with the old 1.0 subsystem.

Which timeout? SSH knows how to handle this error situation. Try 

      ssh -v -s $HOST garbage

The price is not a timeout but a new session key negotiation.

> It is operationally unacceptable to introduce timeouts into the
> protocol version negotiation algorithm.  The current solution in
> 4741bis does not add any additional message flows or session login
> attempts to let the client and server pick the most recent version
> in common.

Now you are mixing things again.

/js

-- 
Juergen Schoenwaelder           Jacobs University Bremen gGmbH
Phone: +49 421 200 3587         Campus Ring 1, 28759 Bremen, Germany
Fax:   +49 421 200 3103         <http://www.jacobs-university.de/>

---

From: Andy Bierman <[email-masked]>
Date: Fri, 03 Sep 2010 09:19:28 -0700

> Non-printable characters would be a bit inconvenient for cut-n-paste.

In fact, this is all that the proposed massive change to the protocol would accomplish.

As Randy pointed out, the problem is that we are using application content
to control framing at the transport layer.  This is fundamentally broken and no
character sequence that the user needs to provide in the content will work
when it comes to preventing a hacker from using this SSH EOM marker to inject
a new message into the channel.

The fact that 0x04 creates invalid XML is irrelevant to this attack.
The fact that a NETCONF server will interpret particular SSH channel content
as an application message framing marker is all that is relevant.

Causing the server to break the byte stream and start a new NETCONF message
is the problem, not the fact that the current EOM is legal XML.
The attack works just fine at injecting a new message with the old marker
or the new one.

QED: it is pointless to make this massive change

Andy

---

From: Ladislav Lhotka <[email-masked]>
Date: Fri, 03 Sep 2010 18:32:55 +0200

> the operator can configure the server to use a non-default port.

It will be pain now, but it will be bigger pain (or even impossible)
later. IMO the problem is that it is really broken. Consider this:

<rpc message-id="101" foo="]]>]]>"
     xmlns="urn:ietf:params:xml:ns:netconf:base:1.0">
  <my-own-method xmlns="http://example.net/me/my-own/1.0">
    <my-first-parameter>14</my-first-parameter>
    <another-parameter>fred</another-parameter>
  </my-own-method>
</rpc>

It is a well-formed XML document, and valid according to 4741bis -
moreover, the server is required to return the "foo" attribute. Yet it
will most likely stymie the server.

Lada 

-- 
Ladislav Lhotka, CESNET
PGP Key ID: E74E8C0C

---

From: Andy Bierman <[email-masked]>
Date: Fri, 03 Sep 2010 09:57:03 -0700

> will most likely stymie the server.

According to 4742, the server will break this octet stream into 2 invalid NETCONF messages.
The NETCONF protocol layer will not see this as a single message.
There is no useful purpose for the foo attribute in the example above.

We have a valid use-case: worrying about message injection attacks caused
by the SSH EOM sequence.  The WG seemed to decide that shutting down
the session did not completely solve the security problem so there was no
point in doing it (odd that this is not the approach taken for other
types of security).  But that was a document change that actually addressed
the use-case.

IMO, the current ']]>]]>' sequence is unfortunate but it is not a problem
in itself, so 'fixing it' is not necessary.  There are no valid use-cases
for an application needing to send ']]>]]>' to the server.  The application MUST
use '&gt;' instead of '>'.  End of that use-case.

The only real use-case is for a hacker to cause the NETCONF server to incorrectly
interpret this sequence as a real EOM marker. That is the real problem to be solved
here.  I would fully support a solution which addresses the real problem.

Andy

---

From: Ladislav Lhotka <[email-masked]>
Date: Fri, 03 Sep 2010 19:43:08 +0200

On Fri, 03 Sep 2010 09:57:03 -0700, Andy Bierman <[email-masked]> wrote:
> According to 4742, the server will break this octet stream into 2
> invalid NETCONF messages.  The NETCONF protocol layer will not see
> this as a single message.  There is no useful purpose for the foo
> attribute in the example above.

Somebody might want to use it to demostrate that my or your server is a
piece of crap, unable to survive receiving a trivial 4741bis-compliant
message. But this is only simplest case. In another example from
4741bis, one could assume that the value of ex:user attribute is
supplied by the user:

<rpc message-id="101"
     xmlns="urn:ietf:params:xml:ns:netconf:base:1.0"
     xmlns:ex="http://example.net/content/1.0"
     ex:user-id="fred]]>]]>">
  <get/>
</rpc>

> IMO, the current ']]>]]>' sequence is unfortunate but it is not a
> problem in itself, so 'fixing it' is not necessary.  There are no
> valid use-cases for an application needing to send ']]>]]>' to the
> server.  The application MUST use '&gt;' instead of '>'.  End of that
> use-case.

The problem is that a client could normally use an off-the-shelf
validating parser to check the message for validity, which however won't
catch such cases as they are well-formed and valid. So the client would
have to scan the message on its own for the presence of the offending
sequence.

> 
> The only real use-case is for a hacker to cause the NETCONF server to
> incorrectly interpret this sequence as a real EOM marker. That is the
> real problem to be solved here.  I would fully support a solution
> which addresses the real problem.

Even a non-validating XML parser will signal the presence of the 0x04
character *anywhere* in the document.

Lada

-- 
Ladislav Lhotka, CESNET
PGP Key ID: E74E8C0C

---

From: Ladislav Lhotka <[email-masked]>
Date: Fri, 03 Sep 2010 19:45:44 +0200

On Fri, 03 Sep 2010 09:19:28 -0700, Andy Bierman <[email-masked]> wrote:
> As Randy pointed out, the problem is that we are using application
> content to control framing at the transport layer.  This is
> fundamentally broken and no character sequence that the user needs to
> provide in the content will work when it comes to preventing a hacker
> from using this SSH EOM marker to inject a new message into the
> channel.
> 

No, it is not, because invalid XML can be easily detected by a
non-validating parser. This raises the bar for an exploit considrably.

Lada

> The fact that a NETCONF server will interpret particular SSH channel
> content as an application message framing marker is all that is
> relevant.
> 

-- 
Ladislav Lhotka, CESNET
PGP Key ID: E74E8C0C

---

From: Andy Bierman <[email-masked]>
Date: Fri, 03 Sep 2010 10:56:33 -0700

> non-validating parser. This raises the bar for an exploit considrably.

I do think the sequence we picked is going to show up in normal content
in the real world, in real deployments.  Nobody is going to want
the user name 'joe]]>]]>'.  That would break more applications than NETCONF.
Even if they did want this obscure character sequence in content, they
can use a character entity to cause the XML parser to convey the correct
content to the application.  (Try sending ]]&gt;]]&gt; and see that
it does not cause the EOM problem and the application is delivered ']]>]]>').

I think you are mixing up the layers.
The NETCONF-over-SSH server implementation uses this EOM marker to break up SSH channel stream.
The XML parser is never going to see any character sequence that is
a valid EOM marker.  An offline parser that does not simulate correct
NETCONF messages is not a valid use-case.  It is trivial for a tool to
scan the input string for the EOM substring.

Andy

---

From: Ladislav Lhotka <[email-masked]>
Date: Sat, 04 Sep 2010 18:32:28 +0200

> I do think the sequence we picked is going to show up in normal

You probably do NOT think ...

> the user name 'joe]]>]]>'.  That would break more applications than

Crackers might use it in various ingenious ways for attacking the
server. Even if the only result is that the session is killed, it's
still a DoS attack.
 
> I think you are mixing up the layers.

I don't think so. The sequence ']]>]]>' was selected because it was
believed that it cannot appear in the Messages, Operations and Content
Layer, which however is not true - it can appear in any and all of
them. This is where the layers got mixed.  In contrast, 0x04 does have
the desired property.

> scan the input string for the EOM substring.

4741bis doesn't forbid this sequence (yet), so why is an implementor
supposed to know that the client has to perform this scan? It's true
that e.g. xmllint changes ']]>]]>' to ']]&gt;]]&gt; in *attribute
values*, but not in comments and PIs, for example in

<interface>
  <name>eth0</name>
  <description>
    The following processing instruction terminates the NETCONF
    session: <?kill-session ]]>]]>?>
  </description>
</interface>

For standard XML tools this is perfectly valid and they won't touch the
PI, whereas 0x04 immediately breaks the well-formedness.

Lada

-- 
Ladislav Lhotka, CESNET
PGP Key ID: E74E8C0C

---

From: Juergen Schoenwaelder <[email-masked]>
Date: Sat, 4 Sep 2010 18:46:40 +0200

On Sat, Sep 04, 2010 at 06:32:28PM +0200, Ladislav Lhotka wrote:

[...]

Replacing a magic character sequence with some other magic character
sequence does not solve the problem. Code injection is a problem
because clients fail to properly check user input; assuming those
clients will detect the new magic character sequence or validate the
XML before they send stuff to a server is IMHO a broken assumption.
If we want to make the framing robust, we have to move to a length
prefixed chunked encoding. Such an encoding is robust, but not
cut'n'paste friendly and _requires_ that clients do a bit more work.

A typical engineering trade-off between leaving things as they are
(perhaps putting up clear warning signs for client writers that will
nevertheless be ignored) or fixing the problem at the expense of a
transition phase and the loss of easy cut'n'paste debugging. 

Having the cost of a transition phase while only partially solving the
problem (by moving to another magic character sequence) does not seem
to be good engineering to me.

/js

---

From: Andy Bierman <[email-masked]>
Date: Sat, 04 Sep 2010 10:44:38 -0700

> cut'n'paste friendly and _requires_ that clients do a bit more work.

The servers have to do more work as well.
Servers that stream replies to avoid buffering
will be forced to use at least a small buffer.
As if XML wasn't verbose enough, adding lots
of Chunk...Length 512... lines to the reply makes it even worse.

Application message framing for SSH2 does not seem like a
NETCONF-specific problem.

Also, a clever hacker that knows an application is going to used fixed
sized chunks can figure out how much padding needs to be inserted,
along with a new 'chunk' (the injected messages), or the known size can be
used to truncate the current message and insert a new one without changing
the chunk size.

The attack is not stopped by length based chunking, just made a little
more difficult.  If the previous message (the one truncated) does not
cause the session to terminate, then the hacker has succeeded and
the injected message(s) will be processed. It doesn't matter if
the chunks that follow are garbage and terminate the session.

I assume there is no requirement that a chunk match a NETCONF message in size.
That would require a server to fully buffer every reply -- a non-starter.
That also assumes some rather severe implementation requirements on the client.
An application may choose to utilize fixed sized chunks as well.

> to be good engineering to me.

I am not sure the WG entirely agrees on the real problem,
or the real costs of each solution proposal.

Andy

---

From: "Randy Presuhn" <[email-masked]>
Date: Sat, 4 Sep 2010 11:13:38 -0700

Hi -

> From: "Andy Bierman" <[email-masked]>
> To: "Ladislav Lhotka" <[email-masked]>; <[email-masked]>
> Sent: Saturday, September 04, 2010 10:44 AM
...
> the chunks that follow are garbage and terminate the session.

Although this is the attack you keep returning to, this is not the
concern that worries everyone else.  The attacks others are
concerned with are based on using the user input to subvert
the rest of the processing, NOT modification of the stream by someone
in the middle.  The only way user input could subvert length-based
chunking is if the implementation failed to account for the length of
the user input.
 
Leading to your next point...

...

I disagree with this reasoning.  Computing what the length of a reply
will be does not require actually buffering the whole thing.  It will entail
some extra processing cost, but buffering is not logically required.

Randy

---

From: Andy Bierman <[email-masked]>
Date: Sat, 04 Sep 2010 11:30:11 -0700

> some extra processing cost, but buffering is not logically required.

The alternative is extremely expensive -- locking down the instrumentation
or somehow pre-computing the length of the reply without changing any of the
contents of the reply between the time the length is computed and the time
the real reply is generated.

So what real problems are addressed by this solution?
What problems are you trying to solve?  IMO we do not have
consensus on that issue.  We should not even discuss solutions
until there is WG-chair declared consensus on the problems to solve.

Just because the <rpc> message allows attributes
to be added does not mean an application MUST be able to use '>' instead of '&gt;'.
This seems like a minor work-around in an meaningless feature, compared to
the operational and implementation costs of introducing the semi-proposed changes.

Andy

---

From: Andy Bierman <[email-masked]>
Date: Sat, 04 Sep 2010 11:31:35 -0700

> to be good engineering to me.

I think I am confusing the layers again.
A chunking mechanism invented just for this purpose could have
EOM markers and chunk messages, done in some way that made
it difficult to inject new NETCONF operations.  But a hacker does not need
a new message if he can hijack a known 'catch-all' operation like <edit-config>.

>From the bad application's POV, it does not know or care
that the user input is 1000 chars instead of a 'normal' 10.
The unchecked input will simply be thrown into the buffer and
the transport layer will happily encode the chunks that contain the
altered message stream -- allowing unintended modification of datastore contents.

Without a written proposal on NETCONF over SSH message chunking,
and a written proposal defining all the security attacks and other problems
that this protocol change is supposed to solve,
there is no reason to consider any changes to NETCONF, and we should just
move forward with 4741bis/4742bis, and ignore this issue.

Andy

---

From: Juergen Schoenwaelder <[email-masked]>
Date: Sat, 4 Sep 2010 21:16:57 +0200

On Sat, Sep 04, 2010 at 08:30:11PM +0200, Andy Bierman wrote:

> We should not even discuss solutions until there is WG-chair
> declared consensus on the problems to solve.

May I ask the WG chairs to approve that we have a discussion?

/js

---

From: Andy Bierman <[email-masked]>
Date: Sat, 04 Sep 2010 12:41:23 -0700

> May I ask the WG chairs to approve that we have a discussion?

How about a focused discussion?

Those who want to make changes to the document need to explain why first.
Change proposals need to be more detailed than an example in an email.
Operational coexistence with 4741 and 4742 must be explained.

We need to decide if changing the SSH message framing is worth the
cost.  Without agreement on the problems that are being addressed,
and a common detailed understanding of the solution(s), this cannot be done.

Andy

---

From: Juergen Schoenwaelder <[email-masked]>
Date: Sat, 4 Sep 2010 22:37:16 +0200

> and a common detailed understanding of the solution(s), this cannot be done.

Here is an attempt to define the issue that was discussed:

  The End of Message (EOM) marker is valid XML and can thus appear in
  a valid XML document. Simple minded clients, even if they validate
  the XML they send, can produce messages that cause the framing
  protocol to fail. This is in particular a problem if user data is
  passed unchecked into an XML document, leading in the best case to
  framing errors or in the worst case to an easy way to inject
  arbitrary NETCONF messages.

/js

---

From: Andy Bierman <[email-masked]>
Date: Sat, 04 Sep 2010 14:41:04 -0700

>   arbitrary NETCONF messages.

OK -- so even though nobody really wants to use this char sequence, a
client tool may let the current EOM through, but the new one
cannot even be entered into a text buffer, let alone pass an XML parser.

It seems to me that it is useful to be able to open a raw SSH session
with 'ssh -p 830 -s me@myserver netconf' and be able to type or cut-n-paste
XML right to the server from a text file.   IMO that seems like a lot to
give up in order to prevent the SSH EOM from getting passed by a client
unaware of the way RFC 4742 works.

So assuming the WG is changing its mind, and deciding this
is an important problem to solve right now, what are the solution
proposals, and how does each one co-exist (old-c/new-s; new-c/old-s)?

Andy

---

From: Per Hedeland <[email-masked]>
Date: Sun, 05 Sep 2010 14:08:02 +0200

On 2010-09-04 18:46, Juergen Schoenwaelder wrote:
> prefixed chunked encoding.

Sorry, but this is not true. EOM framing can be perfectly robust, as
evidenced by several existing protocols - but to be robust, it must not
assume that the EOM can never appear in content. I hesitate to point to
it, since the exact same discussion that we had 18+ months ago didn't
lead anywhere either, but in
http://www.ietf.org/mail-archive/web/netconf/current/msg04231.html I
described how a variant of the SMTP dot-stuffig logic could be applied
to the currently defined SSH EOM sequence. I don't recall that there
were any technical objections raised, and I still believe that the logic
is correct.

> Such an encoding is robust, but not

I believe that my suggestion is robust *and* cut'n'paste friendly - it
too requires that clients (and servers) do a bit more work, but probably
less than for a length prefixed chunked encoding (at least on the client
side).

--Per Hedeland

---

From: Andy Bierman <[email-masked]>
Date: Sun, 05 Sep 2010 09:37:50 -0700

> less than for a length prefixed chunked encoding (at least on the client

I like your solution the best, although the backward compatibility hit
that any solution would bring is still a big concern to me.

I like it because the cut-and-paste use-case is unharmed.
Anyone needing to generate client messages this way can still do so
as long as the obscure sequence ]]>]]> is not needed in content.

A real application (and every server) will take a performance hit,
and have to maintain state it didn't need before, but the solution
is localized to the same code that would treat this char sequence as an EOM,
which is good.

Instead of changing the EOM, the improbable EOM-as-content is changed, so
in most cases, it is even backward-compatible with 4741 implementations.

Andy

---

From: Ladislav Lhotka <[email-masked]>
Date: Mon, 06 Sep 2010 11:56:13 +0200

On Sun, 05 Sep 2010 14:08:02 +0200, Per Hedeland <[email-masked]> wrote:
> evidenced by several existing protocols - but to be robust, it must
> not assume that the EOM can never appear in content. I hesitate to

Yes, that's the point. The reasoning in 4742bis behind choosing ']]>]]>'
as EOM would look quite funny if the whole truth was told:

    This character sequence cannot legally appear outside of a comment,
    processing instruction or attribute value in an XML document, ...

> point to it, since the exact same discussion that we had 18+ months
> ago didn't lead anywhere either, but in

Let me quote it:

> 1. Transmission of PDU:

> a) Any occurrence of the sequence "]]>]]" is rewritten to "]]>]]]".

The sequence ']]>]]]' can appear in XML as well, even in the element
text (end of a CDATA section followed by three right brackets), so the
backward translation in 2b below could produce wrong results, too,
couldn't it?

Lada

> b) The sequence "]]>]]>" is appended to the PDU.

> 2. Reception of PDU:

> a) The sequence "]]>]]>" indicates the end of the PDU.

> b) Any occurrence of the sequence "]]>]]]" is rewritten to "]]>]]".

> were any technical objections raised, and I still believe that the
> logic is correct.
> 

-- 
Ladislav Lhotka, CESNET
PGP Key ID: E74E8C0C

---

From: Per Hedeland <[email-masked]>
Date: Mon, 06 Sep 2010 12:53:39 +0200

> as EOM would look quite funny if the whole truth was told:

Right - but as Andy pointed out, it doesn't matter whether the EOM can
"legally" appear in content when you're worried about e.g. damage caused
by unvalidated user input. A really robust "EOM-based" transport should
always escape the EOM when it occurs in content, which takes care also
of "illegal" content.

> backward translation in 2b below could produce wrong results, too,

No, per 1a ']]>]]]' would become ']]>]]]]', and then get back to
']]>]]]' per 2b. As I wrote, I didn't invent the logic, just mapped if
from SMTP (though of course I may have made a mistake there).  SMTP
doubles all occurrences of a leading dot on a line (i.e.  "CRLF."  ->
"CRLF..") even though the EOM is actually "CRLF.CRLF" - that's what
makes the dot-stuffing transparent. (There may well be other ways, but
this one is known to work.:-)

--Per

---

From: Ladislav Lhotka <[email-masked]>
Date: Mon, 06 Sep 2010 14:20:18 +0200

> always escape the EOM when it occurs in content, which takes care also

The 0x04 EOM needn't be escaped, any document containing it can be
rejected outright. XML does provide a marker that really cannot appear
in content, so what can be more robust?

> this one is known to work.:-)

Oh yes, sorry, I missed the point that it isn't the entire "]]>]]>"
sequence that is translated. So the only caveat is that the XML document
must not be touched in between by anything else, not even xmllinted or
canonicalized. 

Lada

-- 
Ladislav Lhotka, CESNET
PGP Key ID: E74E8C0C

---

From: Per Hedeland <[email-masked]>
Date: Mon, 06 Sep 2010 15:46:21 +0200

On 2010-09-06 14:20, Ladislav Lhotka wrote:
> rejected outright.

By whom? The concern was that the sender would not properly validate
e.g. user input, and could thus pass a 0x04 through. At the other end
there won't be a document containing 0x04, as it will have been
interpreted as EOM by the transport.

> XML does provide a marker that really cannot appear

A transport that doesn't assume anything about the content. Of course
the sender can compensate for the non-robustness of the transport by
rejecting the 0x04 that the transport can't handle - just as it can do
it by doing ">" -> "&gt;" mapping for the currently defined EOM that the
currently defined transport can't handle. Though I do believe you shot
the latter down in the previous round, in
http://www.ietf.org/mail-archive/web/netconf/current/msg04247.html

> must not be touched in between by anything else, not even xmllinted or

Here's my response to that from the previous round:-) -
http://www.ietf.org/mail-archive/web/netconf/current/msg04252.html

--Per

---

From: Andy Bierman <[email-masked]>
Date: Mon, 06 Sep 2010 08:38:39 -0700

> http://www.ietf.org/mail-archive/web/netconf/current/msg04252.html

The particular EOM sequence is not that important.
The byte-stuffing could be ']]' --> ']]]' and that would work as well.
The longer the EOM sequence, the less likely it will occur in user content.
My previous sugguestion to always translate '>' to '&gt;' works for the same
reason, but will occur the most often.

So an offline validation tool will be OK if it permits the ]]>]]> sequence
in an XML attribute.  The sequence is only changed to ]]>]]]> during
NETCONF message transmission (and back to ]]>]]> by the receiver).

Andy

---

From: Ladislav Lhotka <[email-masked]>
Date: Mon, 06 Sep 2010 18:24:27 +0200

> By whom? The concern was that the sender would not properly validate

By the client. 4741bis requires in sec. 3 that NETCONF messages MUST be
well-formed, so the client MUST catch all illegal characters before
passing them to the transport layer in any case, be they EOM or not.

> http://www.ietf.org/mail-archive/web/netconf/current/msg04247.html

That message was about making escapes blindly, like in "s/>/&gt;/g".
Your method is cleverer than that, of course.

> http://www.ietf.org/mail-archive/web/netconf/current/msg04252.html

Sure, but it's important to keep in mind that the the transported
infoset would be different, e.g. by those who want to copy and paste
NETCONF PDUs.

Lada

-- 
Ladislav Lhotka, CESNET
PGP Key ID: E74E8C0C

---

From: Andy Bierman <[email-masked]>
Date: Mon, 06 Sep 2010 10:05:50 -0700

> infoset would be different, e.g. by those who want to copy and paste

The bytes on the wire are probably encrypted so capturing them in transit
between 2 NETCONF peers is not realistic anyway.  It is not a use-case
within the scope of the standard.

Andy

---

From: Ladislav Lhotka <[email-masked]>
Date: Mon, 06 Sep 2010 19:14:05 +0200

On Sat, 04 Sep 2010 14:41:04 -0700, Andy Bierman <[email-masked]> wrote:
> with 'ssh -p 830 -s me@myserver netconf' and be able to type or
> cut-n-paste XML right to the server from a text file.  IMO that seems
> like a lot to give up in order to prevent the SSH EOM from getting
> passed by a client unaware of the way RFC 4742 works.

The 0x04 EOM does not imply that this method won't work. For instance,
in Emacs the character can be easily entered from the keyboard, it is
diplayed as "^D" and may be copied and pasted like any other character.

In any case, a simple filter could be used to translate any character
sequence of your choice to the proper EOM marker and pipe the output to
ssh.

Lada

-- 
Ladislav Lhotka, CESNET
PGP Key ID: E74E8C0C

---

From: Ladislav Lhotka <[email-masked]>
Date: Mon, 06 Sep 2010 19:40:17 +0200

> reason, but will occur the most often.

The latter wouldn't work because "&gt;" may appear in the input before
the translation, so the translation back to ">" at the server side may be
wrong. 

Lada

-- 
Ladislav Lhotka, CESNET
PGP Key ID: E74E8C0C

---

From: Ladislav Lhotka <[email-masked]>
Date: Tue, 07 Sep 2010 09:24:26 +0200

> XML before they send stuff to a server is IMHO a broken assumption.

I don't agree. Even now, the client must make sure NETCONF messages
contain no XML-illegal characters. It is irrelevant whether this is done
by an off-the-shelf XML parser or by other means, but failing to do so
violates the NETCONF protocol.

Therefore, any of the XML-illegal characters, e.g. 0x04, may be safely
used by the transport layer as the EOM marker.

> to be good engineering to me.

Yes, it means an incompatible transport and is somewhat less friendly,
though not hostile, to copy-and-paste, but otherwise I think it pretty
much solves the issue without imposing any other requirements on
transport layer processing (in particular, escaping anything in the
payload) because 4741bis already requires that the messages be
well-formed.

Lada

-- 
Ladislav Lhotka, CESNET
PGP Key ID: E74E8C0C

---

From: Wes Hardaker <[email-masked]>
Date: Tue, 07 Sep 2010 06:52:49 -0700

>>>>> On Sun, 05 Sep 2010 09:37:50 -0700, Andy Bierman <[email-masked]> said:

AB> I like it because the cut-and-paste use-case is unharmed.
AB> Anyone needing to generate client messages this way can still do so
AB> as long as the obscure sequence ]]>]]> is not needed in content.

I think we've established well that no marker can be treated properly as
a valid "the non-checking client can't insert it" marker in an SSH (or
TLS) transport.

If we assume there will be clients (simple scripts) that don't check the
input, then the server must make a decision:

  1) drop it and accept the next
  2) drop everything

We're trying really really hard to come up with a way so that the server
can do #1, although I don't recall anyone saying why this is a critical
feature to have.  IMHO, dropping the session is simply easier on the
server's resources (though I'm sure that some will argue that having the
client start a new session will tax different resources).

This whole problem still stems from trying to insert a framing layer
between two protocols that don't have it.  And we're trying to do it in
a way that, not surprisingly, doesn't work.  If you asked any protocol
designer "lets just do framing by injecting a magic bullet" you'd find
that no one would accomplish it this way.  They'd do proper
encapsulation via some other *true* packeting layer instead because it
avoids these parsing problems that can't be solved without a proper
framing layer.

In other words, we now have this stack:

  Netconf Engine
  XML-Parser
  SSH Layer
  TCP
  IP

And we're trying to come up with some magic signaling that allows the
XML-Parser (or more likely a small piece of code right before it) to
break up the data received from the SSH layer into chunks to hand to the
real XML-parser.

But no one in their right mind, after studying the problems that we've
documented so well, would do it this way given the choice!  They'd do
the *proper* thing and insert a new layer (like BEEP did) that properly
frames things:

  Netconf Engine
  XML-Parser
  ***Framing Layer
  SSH Layer
  TCP
  IP

A true framing layer would have proper "this is the start of a message
that is X bytes long" markers on it (just like BEEP does).  The only way
to truly tell how long a message will be would be to frame it properly
*in front* of the message, not behind it (or possibly both).  Any other
solution allows for accidental or intentional injection of the magic
marker by a poorly written client (which, as I've mentioned before, is
part of the reason for the protocol: allow for simple clients).

So between these now 3 choices:

  1) drop it and accept the next
     1a) via a magic bullet at the end of the message
     1b) via proper framing in the front of the message
  2) drop everything

I really don't get why we're fighting to take 1a when it's the most
problematic solution on the server's side.  Who cares what happens on
the client side, because we know there will be clients that fail to do
the checks.  So to protect the server, either 1b or 2 is simply better
for the server and isn't subject to errors.

1b makes cut-n-paste harder (though you could do it)

2 means everything is simple, but means the netconf server won't accept
traffic from sessions that contain broken messages (which I've never
thought was necessary anyway).
-- 
Wes Hardaker
Cobham Analytic Solutions

---

From: Andy Bierman <[email-masked]>
Date: Tue, 07 Sep 2010 09:26:44 -0700

>   1) drop it and accept the next

The WG (except for you and me) seems to think this is not
the issue.  The 4741bis text is still silent on this one.
A server can choose either behavior right now.

The issue seems to be that the application needs to provide ]]>]]>
without checking it.  Yet as your list of layers shows, some part
of the client code needs to be aware that it is sending message + EOM
to the server.

> part of the reason for the protocol: allow for simple clients).

Determining the message size (X) requires that the message be generated
and held in memory, or that some very specialized code be written which
never alters the data between size calc and real generation.

Buffering a small request on a host with lots of memory (the client) is not
a big deal -- even when sending a large request (e.g., 4 MB <config> trees).
But buffering the entire reply or locking down the instrumentation and
doing 2 dives per request is not likely to be implemented on a server.

IMO we are good at ignoring real operational problems but focus like a hawk
on the theoretical problems that are not actually showing up in any
real deployments.

I think th byte stuffing works.
Can you explain why it doesn't work?

Andy

---

From: Juergen Schoenwaelder <[email-masked]>
Date: Tue, 7 Sep 2010 19:12:40 +0200

On Tue, Sep 07, 2010 at 03:52:49PM +0200, Wes Hardaker wrote:
 
> They'd do the *proper* thing and insert a new layer (like BEEP did)
> that properly frames things:
> 
>   Netconf Engine
>   XML-Parser
>   ***Framing Layer
>   SSH Layer
>   TCP
>   IP

I believe we actually have this stack - except that the existing SSH
framing layer is not very robust. The magic marker is a NETCONF over
SSH mapping property, it does not exist in the BEEP transport.

Perhaps we should outsource the whole problem to the SSH layer. ;-)

/js

---

From: "t.petch" <[email-masked]>
Date: Tue, 7 Sep 2010 18:30:12 +0200

----- Original Message -----
From: "Wes Hardaker" <[email-masked]>
To: "Andy Bierman" <[email-masked]>
Cc: <[email-masked]>
Sent: Tuesday, September 07, 2010 3:52 PM

> the *proper* thing and insert a new layer (like BEEP did) that properly

I disagree (with Randy too) about the Framing Layer.  This is a common problem
with other SDOs and one I know  regards delineation as part of Adaptation, which
sits between the layers, but is regarded as part of the server.

So for me, Framing is an SSH issue n your first stack, we have it but it does
not work (because I and others did not know XML as well as we thought).

As Per has pointed out, anything can be solved be escaping/bit-stuffing but is
it worth it?  yes to plug significant security holes, probably not for the
ability to resynch after an error, which I see was the reason given in February
2004, when it was issue 5.1.

Incidentally, the X bytes long approach is in syslog; syslog drew an IPR claim
from H***** which the syslog list puzzled over and could not see what it was
about except X bytes long which was generally thought to be prior art.  The
relevant Patent is now published and does indeed turn out to be over X bytes
long (which puts me off X bytes long no matter how much prior art there is).

Tom Petch

---

From: Phil Shafer <[email-masked]>
Date: Tue, 7 Sep 2010 13:00:14 -0400

>A server can choose either behavior right now.

+1 for this is an issue.

>> the *proper* thing and insert a new layer (like BEEP did) that properly

+1 for adding a real framing layer.

>never alters the data between size calc and real generation.

As I've pointed out here many times, chunking allows the client and
server to avoid having to keep the entire message in memory.  I've
even posted detailed descriptions of how this works.  Let's move
past this and deal with the real issues that adding a framing layer
will introduce (which is mainly the backwards compatibility issue).

Thanks,
 Phil

---

From: Wes Hardaker <[email-masked]>
Date: Tue, 07 Sep 2010 10:50:03 -0700

>>>>> On Tue, 07 Sep 2010 09:26:44 -0700, Andy Bierman <[email-masked]> said:

>> 1) drop it and accept the next

AB> The WG (except for you and me) seems to think this is not
AB> the issue.  The 4741bis text is still silent on this one.
AB> A server can choose either behavior right now.

You're right!  I forgot:

  3) Don't worry about interoperability in this case

Certainly a valid path forward.

AB> Determining the message size (X) requires that the message be generated
AB> and held in memory, or that some very specialized code be written which
AB> never alters the data between size calc and real generation.

AB> Buffering a small request on a host with lots of memory (the client) is not
AB> a big deal -- even when sending a large request (e.g., 4 MB <config> trees).
AB> But buffering the entire reply or locking down the instrumentation and
AB> doing 2 dives per request is not likely to be implemented on a
AB> server.

I certainly understand and agree with your point.  Though there are
other ways around that.  You can do "partial lengths" as well, though.
The next 4096 bytes is part of a message.  It will continue after that
so don't process (fully) yet until I tell you when the last part has
arrived.  Again, this still falls under proper framing solutions in a
framing layer.

AB> I think th byte stuffing works.
AB> Can you explain why it doesn't work?

I've seen a number of things you could be referring to as "byte
stuffing", so I'm not sure.
-- 
Wes Hardaker
Cobham Analytic Solutions

---

From: Wes Hardaker <[email-masked]>
Date: Tue, 07 Sep 2010 10:52:13 -0700

>>>>> On Tue, 7 Sep 2010 19:12:40 +0200, Juergen Schoenwaelder <[email-masked]> said:

>> IP

JS> I believe we actually have this stack - except that the existing SSH
JS> framing layer is not very robust. The magic marker is a NETCONF over
JS> SSH mapping property, it does not exist in the BEEP transport.

No, we don't.  We have an attempt to get one layer to "pretend" to be
both a secure tunnel layer (SSH) and a framing layer (]]>]]>).  They're
not separate layers.  If they were, we wouldn't have this issue.

JS> Perhaps we should outsource the whole problem to the SSH layer. ;-)

/me looks in the distance and sees the BEEP folks on their way here
holding a "I told you so" sign.
-- 
Wes Hardaker
Cobham Analytic Solutions

---

From: Wes Hardaker <[email-masked]>
Date: Tue, 07 Sep 2010 10:53:30 -0700

>>>>> On Tue, 7 Sep 2010 18:30:12 +0200, "t.petch" <[email-masked]> said:

tp> Incidentally, the X bytes long approach is in syslog; syslog drew an
tp> IPR claim from H***** which the syslog list puzzled over and could
tp> not see what it was about except X bytes long which was generally
tp> thought to be prior art.  The relevant Patent is now published and
tp> does indeed turn out to be over X bytes long (which puts me off X
tp> bytes long no matter how much prior art there is).

Huh.  So UDP and TCP are subject to patent royalties?  Interesting.
-- 
Wes Hardaker
Cobham Analytic Solutions

---

From: Andy Bierman <[email-masked]>
Date: Tue, 07 Sep 2010 12:07:58 -0700

> will introduce (which is mainly the backwards compatibility issue).

The 1 email example from 8/18 is not that detailed.

A real framing layer (to me) means the content layer cannot possibly
inject false framing messages.

There seems to be 3 proposals to change the framing:
  1) keep ]]>]]>; escape/byte stuff ]]>]] so content layer cannot
     possibly inject a framing message
  2) use 0x4 as the EOM marker; content layer can still insert a framing message but the EOM
     is not valid at all in XML, not even PI and attributes, like ]]>]]>,
  3) use explicit frame-chunk and frame-end messages;  assume the content layer
     cannot insert a framing message that will actually be treated by the other
     side as a framing message

It seems to me only (1) and (3) actually prevent a layer violation.
IMO, (1) is more cut-and-paste friendly than (3), but much more processing is needed.
It is easy to avoid using the EOM as content, and old-way-still-works for (1)
except in rare cases.  Pre-computing lengths requires a script or counting bytes by hand.

Backward compatibility:
 1) use a new SSH subsystem (netconf2)
 2) use the same subsystem (netconf) and the same framing, but just for the <hello>
    - once base:1.1 is established, the new framing is used

The timeout for netconf2 is not as long as if there were no SSH server at all.
The server will respond with an 'unknown subsystem' error quickly.
Relying on some particular NETCONF capability to pick a transport framing
method violates layering.  IMO, (1) is probably better.

Andy

---

From: Juergen Schoenwaelder <[email-masked]>
Date: Tue, 7 Sep 2010 21:19:54 +0200

> not separate layers.  If they were, we wouldn't have this issue.

I disagree. SSH says nothing about ]]>]]>, NETCONF says nothing about
]]>]]>. It is the NETCONF over SSH document that defines the ]]>]]>
framing. There is a slim layer.

> holding a "I told you so" sign.

Sure, and they are right. BEEP is a great piece of design work. But
great design is not a guarantee for wide-spread adoption. BEEP is a
piece of infrastructure lacking a killer application - so nobody
deploys the great piece of infrastructure. What matters at the end is
applications that users find useful.

/js

---

From: Phil Shafer <[email-masked]>
Date: Tue, 7 Sep 2010 15:12:39 -0400

>holding a "I told you so" sign.

If only BEEP worked as well as the folks with signs.  In the great
"which transport will be standard", I was betting on BEEP (multi-channel,
framing, lots of features), but the spec stinks and getting one
implementation to work with another turned out to be a nightmare.
JUNOS supports BEEP, but we do our best to discourage its use.

Thanks,
 Phil

---

From: Ladislav Lhotka <[email-masked]>
Date: Wed, 08 Sep 2010 08:40:13 +0200

>   2) use 0x4 as the EOM marker; content layer can still insert a
>   framing message but the EOM is not valid at all in XML, not even PI
>   and attributes, like ]]>]]>,

... , and hence it is not valid in NETCONF messages (4741bis, sec. 3).

>   3) use explicit frame-chunk and frame-end messages; assume the
>   content layer cannot insert a framing message that will actually be
>   treated by the other side as a framing message
> 

With (2), layer violation is prevented by the NETCONF protocol.

Lada

-- 
Ladislav Lhotka, CESNET
PGP Key ID: E74E8C0C

---

From: Andy Bierman <[email-masked]>
Date: Wed, 08 Sep 2010 01:05:27 -0700

> With (2), layer violation is prevented by the NETCONF protocol.

How?

The 0x4 char will be interpreted by the other side as the framing char,
not as an illegal char in an XML document.  Using (1) or (3), the
NETCONF layer cannot enter the framing sequence.

Andy

---

From: Ladislav Lhotka <[email-masked]>
Date: Wed, 08 Sep 2010 10:56:34 +0200

> NETCONF layer cannot enter the framing sequence.

A NETCONF-compliant client must not send messages that are invalid - and
a message containing 0x04 is invalid at the Message layer. If clients
were authorized to send any rubbish down the raw SSH pipe, no framing
could ever work reliably.

In fact, the 0x04 EOM does exactly what the "]]>]]>" sequence was
supposed to accomplish in 4742 (but failed to). I agree (1) and (3)
methods would work, too, but they imply bigger changes to 4742bis
compared to (2) where the changes would be trivial.

Lada 

-- 
Ladislav Lhotka, CESNET
PGP Key ID: E74E8C0C

---

From: Paul Selkirk <[email-masked]>
Date: Wed, 8 Sep 2010 23:32:00 +0000 (UTC)

Please excuse my appalling ignorance (and my recitation of obvious
facts to the experts), but I'm trying to work out what the problems
are before I can evaluate the solutions.

Parenthetically, and before I even begin this ramble, it would be
really nice if message framing and resynchronization could be
discussed in their own section of 4742bis, rather than as a footnote
to Capabilities Exchange.

NETCONF over SSH uses the magic character sequence "]]>]]>" as an
in-band (from the SSH perspective) end-of-message marker.  SOAP and
BEEP transports don't need this, because they have their own message
framing, but SSH provides only an undifferentiated byte stream, with
no inherent message framing.

This is problematic, because this magic sequence can occur in a well-
formed and validating (but possibly pathological) XML document,
despite the attempted limitations of 4742[bis].  (I think I also saw
mention that it could occur in attributes, although that's not
mentioned in 4742bis.)

Andy succinctly laid out the proposed alternative ways of framing
messages (byte stuffing, 0x04, chunking).

In normal operation, the content is the framing: each request message
begins with <rpc> and ends with </rpc>; similarly for replies.  So the
parser has a perfectly good way to detect message boundaries, without
any magic framing characters.  (Indeed, 4741 section 4 says this
explicitly.)  External message framing only comes into play as a way
(not the only way) of detecting mal-formed or pathological messages.

Regardless of how a problem is detected (framing error or plain old
xml parse error), there is a second, orthogonal issue of how to
resynchronize after an error.  4742[bis] supports (but doesn't
mandate) "soft resynchronization" - drop the unfinished message, and
re-start the parse after the framing sequence.  Or you could do "hard
resynchronization" - close the connection, and force the client to
reinitiate the request.

There is an attack scenario that I confess I don't fully understand -
a simple-minded client blindly passes data from a malicious user, who
deliberately inserts the framing sequence followed by his own message.
Given that the session is over ssh, the client must be authenticated
to the server, and the user must be trusted by the client.  So there's
a failure of the trust model on the client side (out of scope for this
group), exacerbated by a failure to validate outgoing data (an
implementation problem, which can only partially be solved by
strengthening 4741/4742).

If the client has a valid need to send "]]>]]>" as part of a valid
request message (and that's a big if), then something like byte
stuffing would suffice to get it through.  It would require upgrading
the server to understand the stuffing, but any change to the message
framing would require upgrading the client and server together.

This is important - you're talking about requiring a flag day.  If you
control both the server and the client, with software from the same
vendor, you can do that, lucky you.  If you need to interoperate, too
bad.

However, I question the valid need to send "]]>]]>" within a message.
Is there a real use case for it?  If not, and we're just being liberal
in what we accept, it should be rare enough to justify the few extra
cycles spent stuffing and unstuffing.  Also, since it doesn't change
the actual framing sequence, it's the only solution that will continue
to interoperate with un-upgraded clients and servers that don't try to
send the framing sequence within a message.  0x04 and chunking both
require simultaneous upgrades to all clients and all servers, to fix a
problem that very few people should be having.

I would also support hard resynch (close the connection) in the case
of parse errors.  Make the ssh session the framing layer of last
resort.

So okay, how much have I gotten wrong?

				paul

---

From: Juergen Schoenwaelder <[email-masked]>
Date: Thu, 9 Sep 2010 08:07:07 +0200

> (not the only way) of detecting mal-formed or pathological messages.

My understanding of NETCONF is that each message is well-formed XML
document. This means there can legally be some stuff before or after
the <rpc/> element.

> strengthening 4741/4742).

Code injection attacks are a common problem. We can declare this is
always a client failure and move on. Or we can choose to make such
attacks much more difficult. We are elaborating the possibilities.

> of parse errors.  Make the ssh session the framing layer of last

If we keep the framing as is, I prefer to close the connection - but
this drastic measure is really only needed for _some_ of the SSH
transports.

/js

---

From: Phil Shafer <[email-masked]>
Date: Thu, 9 Sep 2010 09:03:24 -0400

Paul Selkirk writes:
>any magic framing characters.

Since </rpc> can also appear inside messages, there's an implicit
requirement (for the ssh transport) that end-of-frame must be
detected as part of xml parsing, which one can view as a layering
violation.  I can't give my xml parser a stream and tell it to read
until some end-of-file condition because the end-of-frame is tied
to the state of the parser.

    <rpc>
        <invoke-rpc-at-time>
            <time>noon</time>
            <invoke>
                <rpc>
                    <reboot/>
                </rpc>
            </invoke>
        </invoke-rpc-at-time>
    </rpc>

The message does not end at the first "</rpc>", and even if we say
"that's not a problem because the inner <rpc> element has to be in
a different namespace (which isn't strictly true if I define the
<invoke> element to accept a NETCONF rpc)", then I need to track
namespaces to detect end-of-frame, which again links the end-of-frame
condition to the state of the xml parser.  It means I need a SAX-like
API instead of a DOM-like one with a callback to read data.

In libxml2 terms, I want to be able to say something like:

    xmlDocPtr docp;
    xmlParserInputPtr stream;
    ...
    stream = netconfSshChunkedInputStream(client);
    ...
    inputPush(ctxt, stream);
    docp = xmlDoRead(ctxt, ...);

where netconfSshChunkedInputStream() (and the callbacks it
stuffs into "stream") hides all the netconf-ness from the
xml parser using an api that hides all the xml-ness from
the stream reader.

(Yes, I can use the SAX-like xmlTextReaderRead() API in libxml2,
but this puts too much of the responsibility for building xml
documents in my hands.  That's what I want the library to do for
me.)

>vendor, you can do that, lucky you.  If you need to interoperate, too

We could define a second ssh subsystem name ("netconf-chunked")
that supports the chunking mechanism I proposed earlier.  The content
is framed based on the length of the chunk with multiple chunks per
message and an explicit end-of-frame marker.  Clients can fallback
to the previous subsystem name ("netconf") if the new one fails.

No flag day is required, since old client use the old/broken ssh
EOM marker, but new clients can support the new scheme.  Eventually
the old mechanism can be deprecated and support can be discontinued.
Given the number of netconf apps currently in the wild, I don't
think this will be a problem.

Thanks,
 Phil

---

From: Wes Hardaker <[email-masked]>
Date: Thu, 09 Sep 2010 07:48:46 -0700

>>>>> On Thu, 9 Sep 2010 08:07:07 +0200, Juergen Schoenwaelder <[email-masked]> said:

JS> If we keep the framing as is, I prefer to close the connection - but
JS> this drastic measure is really only needed for _some_ of the SSH
JS> transports.

It's also in 5539, which suffers from the same problem.

[which is why I think it should go into 4741bis: lets take care of it
properly for all transports]
-- 
Wes Hardaker
Cobham Analytic Solutions

---

From: Juergen Schoenwaelder <[email-masked]>
Date: Thu, 9 Sep 2010 16:58:31 +0200

> properly for all transports]

But that is wrong for those transports that do provide proper framing.

/js

---

From: Paul Selkirk <[email-masked]>
Date: Thu, 9 Sep 2010 16:13:02 +0000 (UTC)

> document. This means there can legally be some stuff before or after

Such as in-band framing.

> attacks much more difficult. We are elaborating the possibilities.

Okay, I didn't know how common the problem was in the real world,
being relatively new to netconf.

We should certainly try to harden the servers, but if the clients are
being so trivially hacked, it really seems like a layer-8 problem -
don't make it so easy on the client side.

> this drastic measure is really only needed for _some_ of the SSH

Is there more than one ssh transport?  Phil talks about "defin[ing] a
second ssh subsystem name".  Or do you mean stream transports in
general (ssh, tls)?

As I understand it (imperfectly), if the framing is trustworthy, then
we can resynchronize to the next framed message.  But the current ssh
framing is not trustworthy (can be maliciously injected), so we have
to resort to closing the session (call it session-based framing).

Also, IIRC, someone pointed out that a code-injection attack could
write out the end of the legitimate message, followed by the framing
sequence, followed by the attack request, and the server would process
it all without error.  So treating a parse error as an attack wouldn't
help, because the server would only find out that things were out of
sync *after* processing the attack request.  So resynchronization is a
bit of a red herring if the framing is untrustworthy.

I think I understand the gravity of the situation now, and I'm on
board with it.

				paul

---

From: Paul Selkirk <[email-masked]>
Date: Thu, 9 Sep 2010 16:22:59 +0000 (UTC)

> to the state of the parser.

And you can't give the parser a stream of bytes, and get back a stream
of documents (DOM trees or the like).

> to the previous subsystem name ("netconf") if the new one fails.

Ah, okay, I thought we were talking about changes within the existing
ssh subsystem.  Clearly I need to read more about how that negotiation
happens.

> think this will be a problem.

True that. :(

Netconf isn't my day job.  I had been hoping to use it for a project,
but gave up because I couldn't find a good, currently active FOSS
implementation.  Now that Andy has released Yuma as open source, I may
revisit that decision.

				paul

---

From: Juergen Schoenwaelder <[email-masked]>
Date: Thu, 9 Sep 2010 19:37:53 +0200

> second ssh subsystem name".  Or do you mean stream transports in

I wrote "SSH transports" but meant "NETCONF transports". Sorry for
that.

/js

---

From: Ladislav Lhotka <[email-masked]>
Date: Thu, 09 Sep 2010 20:24:04 +0200

> API instead of a DOM-like one with a callback to read data.

IMO the transport layer should just check that the message is
well-formed (in the XML sense) and provide a robust framing. So your
broken message above would be accepted and sent by the client's
transport layer. The server will be able to parse it (into a DOM or
whatever), classify it as invalid, return an error message and continue
the session. In contrast, the server should terminate the session after
receiving a message which is not well-formed XML.

Lada

-- 
Ladislav Lhotka, CESNET
PGP Key ID: E74E8C0C

---

From: Andy Bierman <[email-masked]>
Date: Thu, 9 Sep 2010 11:44:57 -0700

Hi,

Phil's example is perfectly valid NETCONF/YANG:

  rpc invoke-rpc-at-time {
    input {
       leaf time { type string; }
       anyxml invoke;
    }
  }

It would be a horrible layer violation to ask the code that
delivers XML instance documents to the XML parser to check namespaces,
XML comments, and PIs to make sure the </rpc> is really the matching
end-tag for the <rpc> that started the NETCONF message.  I would object
strongly to forcing NETCONF peers to parse XML correctly before the 'proper'
XML parser code has examined the byte stream.  In other words, there is no
way that the XML content can be used to decide where each XML instance document
begins and ends.  

Also, chunking or byte-stuffing seems to be the only options
that do not allow the client to inject a false frame marker.  If the client
injects 0x4 and 'other' text in just the right way, then the opposite NETCONF peer
will interpret this 0x4 byte as the border between 2 NETCONF messages,
and just might allow the injected message to be processed.

Andy

-----Original Message-----
From: [email-masked] [mailto:[email-masked]] On Behalf Of Ladislav Lhotka
Sent: Thursday, September 09, 2010 11:24 AM
To: [email-masked]
Subject: Re: [Netconf] how to handle XML parse errors

Lada

-- 
Ladislav Lhotka, CESNET
PGP Key ID: E74E8C0C

---

From: Ladislav Lhotka <[email-masked]>
Date: Thu, 09 Sep 2010 21:15:05 +0200

> way that the XML content can be used to decide where each XML instance document

A *non-validating* XML parser is no rocket science and 4741(bis) already
demands that the messages must be well-formed. If you don't want to
guarantee well-formedness as a bare minimum for interoperability, then I
don't know why bother with all the XML complexity in the first place.
 
> and just might allow the injected message to be processed.

The client's transport layer must detect the byte as part of the
well-formedness check, the 0x04 byte simply MUST NOT be in what pretends
to be a NETCONF message. With the framing methods (1) and (3) in your
classification, the transport layer must also perform certain functions,
otherwise the protocol won't work.

Lada

-- 
Ladislav Lhotka, CESNET
PGP Key ID: E74E8C0C

---

From: Wes Hardaker <[email-masked]>
Date: Fri, 10 Sep 2010 09:29:03 -0700

>>>>> On Thu,  9 Sep 2010 16:13:02 +0000 (UTC), Paul Selkirk <[email-masked]> said:

PS> Also, IIRC, someone pointed out that a code-injection attack could
PS> write out the end of the legitimate message, followed by the framing
PS> sequence

Unless the framing was done before hand via length-encoded or chunking based.
-- 
Wes Hardaker
Cobham Analytic Solutions

---

From: "t.petch" <[email-masked]>
Date: Fri, 10 Sep 2010 17:43:40 +0200

----- Original Message -----
From: "Juergen Schoenwaelder" <[email-masked]>
To: "Wes Hardaker" <[email-masked]>
Cc: <[email-masked]>
Sent: Thursday, September 09, 2010 4:58 PM

> > >>>>> On Thu, 9 Sep 2010 08:07:07 +0200, Juergen Schoenwaelder
> But that is wrong for those transports that do provide proper framing.

Agreed

It is the job of the transport to delineate its client traffic units, a part of
the job of Adaptation ie 4742bis.

Tom Petch