Subject: [Netconf] how to handle XML parse errors

From: Martin Bjorklund <[email-masked]>
Date: Thu, 12 Aug 2010 14:55:24 +0200 (CEST)

Hi,

In Maastricht, we discussed what the server should do if it receives
broken XML.

I suggest the following change:

OLD:

   All NETCONF messages MUST be well-formed XML, encoded in UTF-8.

NEW:

   All NETCONF messages MUST be well-formed XML, encoded in UTF-8.  If
   a peer receives a message that is not well-formed XML, it MUST
   close the NETCONF session.

Comments?

/martin

---

From: Wes Hardaker <[email-masked]>
Date: Fri, 13 Aug 2010 06:49:54 -0700

>>>>> On Thu, 12 Aug 2010 14:55:24 +0200 (CEST), Martin Bjorklund <[email-masked]> said:

MB> NEW:

MB> All NETCONF messages MUST be well-formed XML, encoded in UTF-8.  If
MB> a peer receives a message that is not well-formed XML, it MUST
MB> close the NETCONF session.

MB> Comments?

Count me in for "I like it".  I think this is a much safer approach to parsing.
-- 
Wes Hardaker
Cobham Analytic Solutions

---

From: Andy Bierman <[email-masked]>
Date: Fri, 13 Aug 2010 13:09:09 -0700

> Count me in for "I like it".  I think this is a much safer approach to parsing.

I do not like it.  It is too fragile.
This was not the intent of the WG when 4741 and 4742 were written.
The intent was that the message framing sequence used in each of the
transports would allow a server to resynch to the next XML document
in the input stream.

If the server cannot resynch and understand that a new XML document
is starting, then it MUST shut down the session.  Otherwise, it MAY
send the appropriate rpc-error and continue instead of shutting down the session.

Andy

---

From: Andy Bierman <[email-masked]>
Date: Sun, 15 Aug 2010 11:06:30 -0700

>> Count me in for "I like it".  I think this is a much safer approach to parsing.

OK -- I see what Wes is trying to do -- close the security hole
that exists, because the 'EOM' marker in SSH is actually allowed in content,
so it is an unsafe marker.  Any attempt by a hacker to truncate the message
in progress and inject a new one will fail because the truncated
message will cause the session to terminate.  (Unless the hacker is clever
enough to truncate the current message in a way that is valid XML,
which is completely possible, but much harder to do.)

Applications which are buggy and forget an end tag (for example)
should probably be fixed ASAP and not allowed to continue.

So I remove my objection, and that means there should be no open issues
and 4741bis can go to the IESG right away.

Andy

> _______________________________________________
> Netconf mailing list
> [email-masked]
> https://www.ietf.org/mailman/listinfo/netconf
> 
>

---

From: Kent Watsen <[email-masked]>
Date: Mon, 16 Aug 2010 10:45:25 -0700

AB> So I remove my objection, and that means there should be
AB> no open issues and 4741bis can go to the IESG right away.

I may be missing the subtleties in this, but it seems that closing the session is a bit heavy handed.

First, I don't view this as a security issue.  If security is a concern, then a proper transport should be used (i.e. SSH, TLS, etc.).

Second, the proposed modification says "if a peer receives...", but it seems that more appropriate actions can account for which side of the connection the peer is on.  For instance, a server can return an rpc-error while a client might execute the RPC again, assuming that the RPC is safe and/or idempotent...

Thanks,
Kent

---

From: "Randy Presuhn" <[email-masked]>
Date: Mon, 16 Aug 2010 11:16:27 -0700

Hi -

> From: "Kent Watsen" <[email-masked]>
> To: "Andy Bierman" <[email-masked]>; "Wes Hardaker" <[email-masked]>
> Cc: <[email-masked]>
> Sent: Monday, August 16, 2010 10:45 AM
> Subject: Re: [Netconf] how to handle XML parse errors
...
> First, I don't view this as a security issue.  If security is a concern, then a proper
> transport should be used (i.e. SSH, TLS, etc.).
> 
> Second, the proposed modification says "if a peer receives...", but it seems
> that more appropriate actions can account for which side of the connection
> the peer is on.  For instance, a server can return an rpc-error while a client
> might execute the RPC again, assuming that the RPC is safe and/or idempotent...

If the bad XML was generated by software, there is little hope that re-trying
the operation might somehow produce a different result.  The only use case
I can think of where it would make sense to try to second-guess broken XML
by attempting to re-synchronize would be where there is a human at the
other end of the connection, manually typing in Netconf PDUs.

Supporting that use case would have been in keeping with the spirit of the
discussion at the IAB workshop, but as the Netconf protocol has evolved
it seems like it might be more trouble than it would be worth.

Randy

---

From: Wes Hardaker <[email-masked]>
Date: Mon, 16 Aug 2010 12:51:23 -0700

>>>>> On Sun, 15 Aug 2010 11:06:30 -0700, Andy Bierman <[email-masked]> said:

AB> OK -- I see what Wes is trying to do -- close the security hole that
AB> exists, because the 'EOM' marker in SSH is actually allowed in
AB> content, so it is an unsafe marker.

FYI, I agree that was one and the primary goal in my mind.  After
thinking about it a lot since that original issue came up, I'm not sure
it's ever wise to accept a series of management requests over a stream
when one in the middle is broken.  If you get three messages A, B, and C
and can't parse B it's certainly likely that C shouldn't be executed if
it is highly dependent on the message B succeeding.

Yes, the manager SHOULD wait for a positive response before sending C
but in many networks this may not be easily possible (see the DTN RG for
examples).

---

From: Kent Watsen <[email-masked]>
Date: Mon, 16 Aug 2010 13:12:01 -0700

I'm wondering if this is an academic exercise or if there is an actual problem being solved here.  I'm having a hard time believing that an attacker could stage a successful man in the middle when the transport is either SSH or SSL/TLS...and, if they were that sophisticated, surely they would terminate the XML correctly.

Is this only an issue for when the transport is not encrypted?

Thanks,
Kent

---

From: Juergen Schoenwaelder <[email-masked]>
Date: Mon, 16 Aug 2010 23:17:44 +0200

On Mon, Aug 16, 2010 at 10:12:01PM +0200, Kent Watsen wrote:

> I'm wondering if this is an academic exercise or if there is an
> actual problem being solved here.  I'm having a hard time believing
> that an attacker could stage a successful man in the middle when the
> transport is either SSH or SSL/TLS...and, if they were that
> sophisticated, surely they would terminate the XML correctly.
> 

I agree that a man-in-the-middle scenario or protection of unsecure
transports is not a convincing argument.

My understanding is that the issue is more related to injection
attacks where a client is accepting user input and not properly doing
the XML encoding with the effect that a user can somehow inject an end
of message marker followed by an XML snipped for a subsequent NETCONF
operation. If this happens, the user might do things the client should
have prevented and the client has of course failed to do its job
properly. There is some empirical evidence that web apps have a
tendency to sometimes not properly check user data (e.g. see all the
SQL injection attacks).

Of course, terminating the session is rather drastic if someone tries
to use NETCONF in a manual cut'n'paste mode. And if a nasty user is
smart enough to figure out how to make the client still finish a valid
message with code injection, then we are in bad shape anyway.

These are the arguments I think I have seen and understood. Am I
missing something?

/js

PS: I did not list Wes' argument for pipelined sequences of
    operations. If a client sends multiple operations and does not
    care about error checking, the client gets what it asked for.
    There are many reasons why operations can fail.

-- 
Juergen Schoenwaelder           Jacobs University Bremen gGmbH
Phone: +49 421 200 3587         Campus Ring 1, 28759 Bremen, Germany
Fax:   +49 421 200 3103         <http://www.jacobs-university.de/>

---

From: Kent Watsen <[email-masked]>
Date: Mon, 16 Aug 2010 16:06:57 -0700

Copy-n-pasting XML is a likely scenario - I do it all the time!  And I many times mess it up too  :)  I still think an rpc-error is more appropriate, perhaps one of these?

   error-tag:      missing-element
   error-type:     protocol, application
   error-severity: error
   error-info:     <bad-element> : name of the missing element
   Description:    An expected element is missing.

   error-tag:      bad-element
   error-type:     protocol, application
   error-severity: error
   error-info:     <bad-element> : name of the element w/ bad value
   Description:    An element value is not correct; e.g., wrong type,
                   out of range, pattern mismatch.

   error-tag:      unknown-element
   error-type:     protocol, application
   error-severity: error
   error-info:     <bad-element> : name of the unexpected element
   Description:    An unexpected element is present.

Regarding the concern for users copy-n-pasting multiple pipelined requests, I think there may be a statement about pipelining requests in general, that they SHOULD NOT depend on each other and that they SHOULD be "safe" and "idempotent" (as defined by the HTTP spec)

Thanks,
Kent

---

From: Wes Hardaker <[email-masked]>
Date: Tue, 17 Aug 2010 07:13:51 -0700

>>>>> On Mon, 16 Aug 2010 23:17:44 +0200, Juergen Schoenwaelder <[email-masked]> said:

JS> PS: I did not list Wes' argument for pipelined sequences of
JS> operations. If a client sends multiple operations and does not
JS> care about error checking, the client gets what it asked for.
JS> There are many reasons why operations can fail.

There are "huge delayed networks" where round trips are too long to
wait.  See "Interplanetary Internet" for examples.  Though there are
real-world on-the-ground-now examples as well (usually in "disrupted
environments" like wireless mesh networks).
-- 
Wes Hardaker
Cobham Analytic Solutions

---

From: Wes Hardaker <[email-masked]>
Date: Tue, 17 Aug 2010 07:18:52 -0700

You know...  I hear this argument all the time, and it's a good one.
But you notice we also design all our protocols such that streamlining
is allowed.  If we want to discourage or say "don't do this" why
wouldn't we put a magic cookie in each server's response message that
the management station must copy into the next "request" message to
prove they at least received the last response.  That way they can't be
"stupid" like you say above.  I suspect you'd argue against this "user
protection feature" saying "it doesn't allow for pipelining" :-)

---

From: Juergen Schoenwaelder <[email-masked]>
Date: Tue, 17 Aug 2010 16:30:46 +0200

> protection feature" saying "it doesn't allow for pipelining" :-)

I wrote "the client gets what it asked for". If a client pipelines a
sequence of protocol operations, it needs to be prepared to clean up
any mess that might have been created due to a failure somewhere in
the pipeline. I did not use words such as "stupid" or "don't do this".

I also note that all existing specified NETCONF transports run over
TCP. If your goal is to run NETCONF over DTNs where TCP is not an
option, there probably needs some additional work to be done anyway.

/js

---

From: Wes Hardaker <[email-masked]>
Date: Tue, 17 Aug 2010 07:38:21 -0700

>>>>> On Mon, 16 Aug 2010 13:12:01 -0700, Kent Watsen <[email-masked]> said:

KW> I'm wondering if this is an academic exercise or if there is an
KW> actual problem being solved here.

The issues are operational ("pipelining" as Juergen labeled it) and
security related.

The security related issue is just to protect the agent (where the
important data is stored) from badly coded managers.

The netconf agent should always be what we're trying to protect against
damage, since that's where all the important config is.  By mandating
that agent's "continue processing whenever possible" seems to me like a
bad decision that just leads to errors.

SSH (and other protocols) don't provide proper message framing.  So the
RFC inserts a magic ]]>]]> bullet that can be used to detect the "start
of the next message" so that if the current one can't be parsed you just
skip to the next.  That seems nice until you realize how many operators
are going to write scripts out there for use with netconf.  I'm *sure*
there will be ones written that fail to account for tricky user-input.
I'm sure of it because it's happened repeatedly for decades.

I don't see the goal in making it easier for them while at the same time
making coding the agent harder.  If the manager sends a bad management
request, why *should* the agent resources be used to try and figure out
where in the parsing stream the problem might be.  I do agree that
having the agent return a "error message" before closing the stream
sounds like an excellent idea.  But I don't see the benefit in wasting
the agent's resources trying to recover from a parser error.  Let the
management station figure it out.

In the SSH case (and now TLS I believe), injection attacks become very
very powerful for any management operation that doesn't very carefully
check it's input.

  #!/bin/csh
  echo "please enter your desired Account Name:"
  set loginname=$<

  echo "please enter your Full Name:"
  set name=$<

  cat << EOF | netconf-client remotehost.example.com
  <rpc message-id="101" xmlns="urn:ietf:params:xml:ns:netconf:base:1.0">
    <edit-config>
      <target> <running/> </target>
      <config>
        <top xmlns="http://example.com/schema/1.2/config">
          <user>
            <login>$loginname</login>
            <fullname>$name</fullname>
          </user>
        </top>
      </config>
    </edit-config>
  </rpc>

The above works great if the user is nice.  I'd even bet that most
people *would* check the $loginname field, since that's an obvious place
where bad data would matter.  But I'd bet that many operators will fail
to check the $name field, because why would it matter?  A name can
contain just about anything.  It works less great if the user types in a
fullname of

  ]]>]]>
  <rpc ...><edit-config>...
  <change-password login="root">newsecret</change-password>
  ...

Andy's right, it's still possible to "maybe" complete the XML anyway if
that input isn't checked.  But it takes a lot more work trying to
"guess" what the end of the XML looks like when you can't see the stream
itself to figure out how the management app is encoding the request over
an encrypted stream.  On the other-hand, the existence of the ']]>]]>'
identifier mean the SSH and TLS streams make it very very easy to inject
bad data into a message and immediately start into the next message
controlled by the attacker.  No guessing required.

Closing the session will fix this.
Closing the session will make pipelining actually safer to use as well.

Win-win.

---

From: Ladislav Lhotka <[email-masked]>
Date: Tue, 17 Aug 2010 17:41:13 +0200

>   <change-password login="root">newsecret</change-password>

Even if the session is terminated on non-well-formed input, it's only
little extra work for the bad guy to make it well-formed:

  Black Hat</fullname>...</rpc>]]>]]>
  <rpc ...><edit-config>...
  <change-password login="root">newsecret</change-password>

So terminating the session doesn't really close this security hole.

Lada

-- 
Ladislav Lhotka, CESNET
PGP Key ID: E74E8C0C

---

From: Wes Hardaker <[email-masked]>
Date: Wed, 18 Aug 2010 08:02:28 -0700

>>>>> On Tue, 17 Aug 2010 16:30:46 +0200, Juergen Schoenwaelder <[email-masked]> said:

JS> I wrote "the client gets what it asked for".

Fair enough...

JS> I also note that all existing specified NETCONF transports run over
JS> TCP. If your goal is to run NETCONF over DTNs where TCP is not an
JS> option, there probably needs some additional work to be done anyway.

I generally don't like writing application protocols that "assume TCP is
in use".  It locks us into "managing only certain types of networks",
which I think is a huge mistake.

[I do agree, though, that other work is likely needed too]
-- 
Wes Hardaker
Cobham Analytic Solutions

---

From: Wes Hardaker <[email-masked]>
Date: Wed, 18 Aug 2010 08:03:49 -0700

>>>>> On Tue, 17 Aug 2010 17:41:13 +0200, Ladislav Lhotka <[email-masked]> said:

LL> Black Hat</fullname>...</rpc>]]>]]>
...
LL> So terminating the session doesn't really close this security hole.

No, but it makes it at least require either sniffing an encrypted
connection or having access to exactly what the management station is
sending.  With just a ]]>]]> terminator I can do it blind to any code
that fails to validate every field.  There's a huge difference there.
-- 
Wes Hardaker
Cobham Analytic Solutions

---

From: Juergen Schoenwaelder <[email-masked]>
Date: Wed, 18 Aug 2010 17:37:23 +0200

> which I think is a huge mistake.

Just to clarify this: NETCONF does not require TCP - it just happens
that all existing transports use TCP. The precise requirements are
detailed in section 2 of RFC 4741, here is just a short quote:

   NETCONF is connection-oriented, requiring a persistent connection
   between peers.  This connection must provide reliable, sequenced data
   delivery.

/js

---

From: Ladislav Lhotka <[email-masked]>
Date: Wed, 18 Aug 2010 18:29:56 +0200

> sending.  With just a ]]>]]> terminator I can do it blind to any code

The management station must be sending what's written in the data model,
so the attacker will just need to consult a nicely documented YANG
module and guess a suitable target XML element for his input. It's a
piece of cake.

> that fails to validate every field.  There's a huge difference there.

While we both agree that the server should close the session upon
receiving a malformed message, it must not be advertized as a remedy
against code injection. Failing to validate every field will remain a
huge security hole.

Lada

-- 
Ladislav Lhotka, CESNET
PGP Key ID: E74E8C0C

---

From: Wes Hardaker <[email-masked]>
Date: Thu, 19 Aug 2010 09:43:29 -0700

>>>>> On Wed, 18 Aug 2010 18:29:56 +0200, Ladislav Lhotka <[email-masked]> said:

LL> While we both agree that the server should close the session upon
LL> receiving a malformed message

I think that's all we *need* to agree on, so I'll stop my side of the
argument :-)
-- 
Wes Hardaker
Cobham Analytic Solutions