Subject: Mandatory TLS == OpenSSL everywhere? !?!

From: "Adrien de Croy" <[email-masked]>
Date: Thu, 14 Nov 2013 21:35:00 +0000

Hi all

one of the things that has been troubling me about the mandatory TLS 
discussion, that I don't recall having seen discussed here is the issue 
of the implicit assumption that it's free or low-cost to include TLS 
into a product due to the availability of open source implementations.

I think that assumption needs looking into a bit further.

I'm going to go out on a limb here, and speculate, that if TLS were to 
become mandatory in HTTP/2 that the vast majority of implementers would 
choose OpenSSL for the TLS implementation.

That immediately raises a number of issues.

1. Homogeneity and therefore susceptibility to exploits having 
effectively global reach.
2. Maintainability of the source

A related issue is the complexity argument which I also haven't seen put 
here.

TLS is complex.  The crypto is complex.  Implementations of TLS with the 
cipher suites contain at least many hundreds of thousands of lines of 
code.  Most of it fairly impenetrable from what I've seen sorry to say.

Compared with an implementation of an HTTP stack, a TLS implementation 
completely dwarfs it for complexity and amount of code.   Since these 
things are not maintained by infallible aliens but us mere mortals, 
there will be bugs.

OpenSSL in my experience has had many troubles with vulnerabilities and 
stability.  And it's an enormous undertaking for someone to fathom the 
code to attempt any maintenance on it at all.

Does this mean, that if we made TLS mandatory, that effectively we would 
be placing the security of the web in the hands of the OpenSSL 
contributors.  I think it effectively does.

This is an EXTREMELY disturbing thought.   The security of the systems 
associated with maintenance and deployment of OpenSSL are not at a 
sufficient level to warrant this level of global reliance.  My own 
experience with OpenSSL has not been without serious problems, and in 
fact we've looked to ditch it many times and may still do.

I don't think it is realistic to expect that http agent (server or 
client) developers will put much effort at all into maintenance of the 
TLS library.  So maintenance will remain the domain of the few 
contributors.

Too many eggs in too few baskets.

Adrien

---

From: Tao Effect <[email-masked]>
Date: Thu, 14 Nov 2013 16:58:49 -0500

Yeah... thanks for bringing that up.

The sad thing is that OpenSSL is already basically everywhere (from what I can tell).

I think it might be in all the major browsers, is that correct?

And it's in Apache (via mod_ssl), and most of the other servers that exist, is that correct too?

I'd like to see a pie chart of OpenSSL usage in the top used web servers and top used browsers. Anyone know of one?

Then there's the "OpenSSL is written by monkeys" problem:

http://www.peereboom.us/assl/assl/html/openssl.html

The situation appears to be that we're using a crypto library, written (allegedly) by monkeys, in C, and only a handful of people and/or monkeys have actually looked at the code.

Additionally, C, and just about all its variants, makes it remarkably easy to write insecure code (by accident), and easy to write malicious code that can sit right in the open and not be noticed those working with the code:

https://en.wikipedia.org/wiki/Underhanded_C_Contest

It's not a new problem though... it's been with us "since like forever." You've probably been pwn'd and haven't realized it. :-p

- Greg

--
Please do not email me anything that you are not comfortable also sharing with the NSA.

On Nov 14, 2013, at 4:35 PM, Adrien de Croy <[email-masked]> wrote:

> Hi all
>  
> one of the things that has been troubling me about the mandatory TLS discussion, that I don't recall having seen discussed here is the issue of the implicit assumption that it's free or low-cost to include TLS into a product due to the availability of open source implementations.
>  
> I'm going to go out on a limb here, and speculate, that if TLS were to become mandatory in HTTP/2 that the vast majority of implementers would choose OpenSSL for the TLS implementation.
>  
> 1. Homogeneity and therefore susceptibility to exploits having effectively global reach.
> A related issue is the complexity argument which I also haven't seen put here.
>  
> TLS is complex.  The crypto is complex.  Implementations of TLS with the cipher suites contain at least many hundreds of thousands of lines of code.  Most of it fairly impenetrable from what I've seen sorry to say.
>  
> Compared with an implementation of an HTTP stack, a TLS implementation completely dwarfs it for complexity and amount of code.   Since these things are not maintained by infallible aliens but us mere mortals, there will be bugs.
>  
> OpenSSL in my experience has had many troubles with vulnerabilities and stability.  And it's an enormous undertaking for someone to fathom the code to attempt any maintenance on it at all.
>  
> Does this mean, that if we made TLS mandatory, that effectively we would be placing the security of the web in the hands of the OpenSSL contributors.  I think it effectively does. 
>  
> This is an EXTREMELY disturbing thought.   The security of the systems associated with maintenance and deployment of OpenSSL are not at a sufficient level to warrant this level of global reliance.  My own experience with OpenSSL has not been without serious problems, and in fact we've looked to ditch it many times and may still do.
>  
> I don't think it is realistic to expect that http agent (server or client) developers will put much effort at all into maintenance of the TLS library.  So maintenance will remain the domain of the few contributors.
>

---

From: "Adrien de Croy" <[email-masked]>
Date: Thu, 14 Nov 2013 22:12:40 +0000

I think it is basically everywhere, but I think we're largely saved by 
not using it everywhere since I believe still most http is not using 
TLS.  IME with WinGate the more we use OpenSSL, the more troubles we 
have, hence the desire to use something else especially since we added 
MITM.

But now we're talking about USING it everywhere.  That's an enormous 
difference.

Discussion about mandatory TLS has seemingly glossed over these sorts of 
issues.

------ Original Message ------
From: "Tao Effect" <[email-masked]>
To: "Adrien de Croy" <[email-masked]>
Cc: "HTTP Working Group" <[email-masked]>
Sent: 15/11/2013 10:58:49 a.m.
Subject: Re: Mandatory TLS == OpenSSL everywhere? !?!
>I'd like to see a pie chart of OpenSSL usage in the top used web 
>servers and top used browsers. Anyone know of one?
>
>The situation appears to be that we're using a crypto library, written 
>(allegedly) by monkeys, in C, and only a handful of people and/or 
>monkeys have actually looked at the code.
>
>Additionally, C, and just about all its variants, makes it remarkably 
>easy to write insecure code (by accident), and easy to write malicious 
>code that can sit right in the open and not be noticed those working 
>with the code:
>
>It's not a new problem though... it's been with us "since like 
>forever." You've probably been pwn'd and haven't realized it. :-p
>
>- Greg
>
>--
>>discussion, that I don't recall having seen discussed here is the 
>>issue of the implicit assumption that it's free or low-cost to include 
>>TLS into a product due to the availability of open source 
>>implementations.
>>
>>become mandatory in HTTP/2 that the vast majority of implementers 
>>would choose OpenSSL for the TLS implementation.
>>
>>TLS is complex.  The crypto is complex.  Implementations of TLS with 
>>the cipher suites contain at least many hundreds of thousands of lines 
>>of code.  Most of it fairly impenetrable from what I've seen sorry to 
>>say.
>>
>>OpenSSL in my experience has had many troubles with vulnerabilities 
>>and stability.  And it's an enormous undertaking for someone to fathom 
>>the code to attempt any maintenance on it at all.
>>
>>Does this mean, that if we made TLS mandatory, that effectively we 
>>would be placing the security of the web in the hands of the OpenSSL

---

From: Mike Belshe <[email-masked]>
Date: Thu, 14 Nov 2013 14:15:40 -0800

One of the things that the IETF does with all new standards is to build
multiple, working implementations.  This was done as part of the TLS
development as well as HTTP.

Rob T could comment further, but I don't believe the windows TLS stack has
much (any?) OpenSSL in it.

It shouldn't be a surprise that there are fewer TLS implementations than
HTTP implementations - complexity aside, HTTP is much larger than TLS
today.  But, if HTTP uses it more, we should see that go up over time.

I disagree that we glossed over these issues.

Mike

On Thu, Nov 14, 2013 at 1:58 PM, Tao Effect <[email-masked]> wrote:

> I'd like to see a pie chart of OpenSSL usage in the top used web servers
> and top used browsers. Anyone know of one?
>
> (allegedly) by monkeys, in C, and only a handful of people and/or monkeys
> have actually looked at the code.
>
> Additionally, C, and just about all its variants, makes it remarkably easy
> to write insecure code (by accident), and easy to write malicious code that
> can sit right in the open and not be noticed those working with the code:
>
> It's not a new problem though... it's been with us "since like forever."
> You've probably been pwn'd and haven't realized it. :-p
>
> - Greg
>
> --
> discussion, that I don't recall having seen discussed here is the issue of
> the implicit assumption that it's free or low-cost to include TLS into a
> product due to the availability of open source implementations.
>
> things are not maintained by infallible aliens but us mere mortals, there
> will be bugs.
>
> stability.  And it's an enormous undertaking for someone to fathom the code
> to attempt any maintenance on it at all.
>
> experience with OpenSSL has not been without serious problems, and in fact
> we've looked to ditch it many times and may still do.
>
> I don't think it is realistic to expect that http agent (server or client)
> developers will put much effort at all into maintenance of the TLS
> library.  So maintenance will remain the domain of the few contributors.
>

---

From: Tao Effect <[email-masked]>
Date: Thu, 14 Nov 2013 17:31:17 -0500

Thanks Mike,

I only have a cursory knowledge of what libraries are used where, so I did some searching and found this nice page:

https://en.wikipedia.org/wiki/Comparison_of_TLS_implementations

More searching brought up lots of results for NSS as being, possibly the second most popular choice? Although again, I have no idea and really would like to see up-to-date numbers.

> I think it might be in all the major browsers, is that correct?

Turns out that it's not correct, as some browsers (it seems) switched to NSS, and some maybe never used OpenSSL (again, not clear on this, would need to research it more).

Here's a list of popular apps that use NSS:

https://en.wikipedia.org/wiki/Network_Security_Services#Applications_that_use_NSS

Happy to see Chrome and Firefox there! ^_^

Also am happy to see that you can setup Apache to use mod_nss instead of mod_ssl.

Cheers,
Greg

On Nov 14, 2013, at 5:15 PM, Mike Belshe <[email-masked]> wrote:

> One of the things that the IETF does with all new standards is to build multiple, working implementations.  This was done as part of the TLS development as well as HTTP.
> 
> Rob T could comment further, but I don't believe the windows TLS stack has much (any?) OpenSSL in it.
> 
> It shouldn't be a surprise that there are fewer TLS implementations than HTTP implementations - complexity aside, HTTP is much larger than TLS today.  But, if HTTP uses it more, we should see that go up over time.
>

---

From: "Adrien de Croy" <[email-masked]>
Date: Thu, 14 Nov 2013 23:02:51 +0000

------ Original Message ------
From: "Mike Belshe" <[email-masked]>
>development as well as HTTP.
So the IETF has built a new from-scratch TLS implementation that is 
freely usable?  What's it called? How can we predict how secure that 
will be?

>has much (any?) OpenSSL in it.
Sure, it's hard to tell whether Schannel is easier to use than OpenSSL 
or not though.  And a quick view through windows update history shows 
enough security patches related to schannel to bolster a complexity 
argument.

Glad to learn about NSS though.  We will look further into that.

Of the other implementations mentioned, only relatively few have a 
license that many commercial software companies could or would use.  
Unless people have previous negative experience with a free license 
implementation, they would probably choose it over a commercial license. 
  Lack of patronage of an implementation would lead to security concerns 
as well.
>
>It shouldn't be a surprise that there are fewer TLS implementations 
>than HTTP implementations - complexity aside, HTTP is much larger than 
>TLS today.  But, if HTTP uses it more, we should see that go up over 
>time.
I don't really understand what you mean here.  You mean HTTP is more 
used than TLS?  I'd agree with that.  You're saying you expect to see 
more implementations?  More heterogeneity is a good thing IMO.

At the moment though I do think OpenSSL would have enormous coverage, 
and changing out TLS library is unlikely to be a simple matter for 
existing OpenSSL users.  So there will be a lot of inertia, and a lot of 
continued use of OpenSSL, if only on the server-side.

>I disagree that we glossed over these issues.
I didn't see them discussed on list, but I may have missed it.

Adrien

>>>issue of the implicit assumption that it's free or low-cost to 
>>>include TLS into a product due to the availability of open source 
>>>implementations.
>>>
>>>I'm going to go out on a limb here, and speculate, that if TLS were 
>>>to become mandatory in HTTP/2 that the vast majority of implementers 
>>>the cipher suites contain at least many hundreds of thousands of 
>>>lines of code.  Most of it fairly impenetrable from what I've seen 
>>>sorry to say.
>>>
>>>Compared with an implementation of an HTTP stack, a TLS 
>>>implementation completely dwarfs it for complexity and amount of 
>>>code.   Since these things are not maintained by infallible aliens 
>>>but us mere mortals, there will be bugs.
>>>
>>>and stability.  And it's an enormous undertaking for someone to 
>>>fathom the code to attempt any maintenance on it at all.
>>>
>>>This is an EXTREMELY disturbing thought.   The security of the 
>>>systems associated with maintenance and deployment of OpenSSL are not 
>>>at a sufficient level to warrant this level of global reliance.  My 
>>>own experience with OpenSSL has not been without serious problems, 
>>>and in fact we've looked to ditch it many times and may still do.
>>>
>>>client) developers will put much effort at all into maintenance of 
>>>the TLS library.  So maintenance will remain the domain of the few 
>>>contributors.
>>>

---

From: Geoff Beier <[email-masked]>
Date: Thu, 14 Nov 2013 20:04:09 -0500

> contributors. I think it effectively does.

libcurl, a popular http library, can use 9 different TLS libraries.

They publish a nice comparison chart here:

http://curl.haxx.se/docs/ssl-compared.html

I do suspect that most people who use libcurl for http support use 
whichever TLS library is available in their development environment by 
default, which will be OpenSSL or Secure Channel for the vast majority.

Geoff

---

From: "Nicolas Mailhot" <[email-masked]>
Date: Fri, 15 Nov 2013 07:20:12 +0100

> Glad to learn about NSS though.  We will look further into that.

nss is viable client-side

Server-side? Trying to use mod_nss in apache is an exercise in
frustration. There are too many functional holes compared to mod_ssl. Even
Red Hat does not try and they bought the whole dev team from Netscape with
the directory implementation.

-- 
Nicolas Mailhot

---

From: Phillip Hallam-Baker <[email-masked]>
Date: Sat, 16 Nov 2013 00:04:44 -0500

> much (any?) OpenSSL in it.

I am pretty certain it predates OpenSSL by a very long time. CryptoAPI
certainly predates Netscape.

Its not just the SSL library, it is the crypto library and the X.509 stack.
These are all mix and match. What has tended to happen is that crypto
libraries have been extended to add PKIX and S/MIME and TLS and DTLS have
been natural additions.

-- 
Website: http://hallambaker.com/