Subject: Re: [Curdle] Client-side SSH_MSG_EXT_INFO: Use it or lose it principle!

From: Ron Frederick <[email-masked]>
Date: Sat, 2 May 2020 23:14:28 -0700

On Apr 26, 2020, at 10:33 PM, denis bider <[email-masked]> wrote:
> Awesome!
> 
> Yes, the idea of "global-requests-ok" is not to make the client do anything different. It's to reassure the server that the client won't violate the spec if the server sends a global request.
> 
> The main motivating usage case is so the server can feel at liberty to send the global request "[email-masked] <mailto:[email-masked]>" after user authentication:
> 
> https://cvsweb.openbsd.org/src/usr.bin/ssh/PROTOCOL?annotate=HEAD <https://cvsweb.openbsd.org/src/usr.bin/ssh/PROTOCOL?annotate=HEAD>
> This is essential functionality which allows clients and servers to change host keys without requiring administrative intervention at each client installation.
> 
> Without "global-requests-ok", the server needs a database of client version strings to which it's safe to send the "hostkeys" global request. Otherwise, it can't send it because a bunch of half-baked clients (I'm counting 7 in our list) are borked and can't handle receiving a global request. Or they can handle it usually, but not when they're expecting a response to a channel open.

Yeah, understood. So far, I haven’t implemented “hostkeys" support in AsyncSSH. I may do that at some point, but I’ll probably just do the new host key validation in AsyncSSH but leave it up to the application code to decide if it actually wants to update the list of known_hosts or not (and if so, where). Right now, AsyncSSH will read from ~/.ssh/known_hosts by default if you don’t specify another source, but it never actually modifies that file. In fact, most of the ways of specifying known_hosts in AsyncSSH are focused on providing the set of hosts dynamically, or even allowing applications to provide their own logic to determine whether a specific host key should be trusted or not for a given host/port, so I’d want to do the same thing with reporting new host keys a server advertises.

Since I don’t have this support yet, I haven’t needed to worry about which clients can properly handle global requests or not. I do use global requests in both directions as part of the implementation of "[email-masked] <mailto:[email-masked]>”, but so far I haven’t run into any issues with that. That’s disabled by default, though, so perhaps people are only enabling with peers that don’t have issues with global requests.

> Meanwhile I've received more info about the implementation that disconnects on receiving EXT_INFO from the client. The bug is unfortunately in a widely used Java SSH library. The information I received suggests it's going to be fixed, but I'm afraid there are already a fair number of deployed servers that have the issue.

Interesting. Does it fail on ANY global request being sent from a server, or is it only at certain points that it doesn’t handle these messages correctly?

> While I'm wishing things - I would also like the "delay-compression" extension to be more widely supported because it's the only delayed-compression mechanism that (1) does not have a race condition by design ([email-masked] <mailto:[email-masked]>), and (2) does not require a second key exchange after user authentication (PuTTY mitigation).

Is RFC 8308 the right reference for this option? Are there any implementations of this so far other than from Bitvise?

Also, I’m not familiar with the PuTTY issue around needing a second key exchange. Is that documented somewhere?
-- 
Ron Frederick
[email-masked]

---

From: denis bider <[email-masked]>
Date: Sun, 3 May 2020 18:11:26 -0500

> I do use global requests in both directions as part of the implementation
> of "[email-masked]”, but so far I haven’t run into any issues with
that.
> That’s disabled by default, though, so perhaps people are only enabling
> with peers that don’t have issues with global requests.

Features that are disabled by default end up being used in a small fraction
of connections in practice - I think it's not wrong to guess 1%. Problems
with them are further unlikely to be reported because the user assumes the
problem they experience is why the feature is disabled anyway.

I think the "[email-masked]" mechanism is highly valuable. I would
like to see it more widely supported and enabled by default. Last I
checked, it was disabled by default, no doubt due to cherished clients that
disconnect when they receive it.

I would very much like it to be enabled by default, especially if the
client sends the "global-requests-ok" extension. Bitvise SSH Server enables
"hostkeys" if the client does this. If not, then we use built-in
compatibility rules based on the SSH version string to guess if it's safe
to send global requests.

> Does it fail on ANY global request being sent from a server, or is it only
> at certain points that it doesn’t handle these messages correctly?

The bug I was discussing here is that when operating as a server, the
Maverick implementation (and therefore servers that use it) has a bug where
it crashes if the client sends SSH_MSG_EXT_INFO. There is only one point in
the connection where the client is permitted to send SSH_MSG_EXT_INFO. This
is immediately after the client's first SSH_MSG_NEWKEYS.

Perhaps you mean to ask about clients which fail upon receiving global
requests from SSH servers? These fall mainly into two types:

(1) Clients that disconnect on receipt of ANY global request at any time.
These are outright negligent implementations: RFC 4254 requires the global
requests to be handled and responded to if "want reply" is true. You can't
send an active keep-alive to this kind of client.

(2) Clients that disconnect when in a special state, specifically when
expecting a channel open confirmation from the server. These are oversights
that were generally fixed in later versions of the software, but people
still use earlier versions that had the bug. When dealing with this type of
client, it's almost-safe to send an active keep-alive (very low chance of
client being in the crash-susceptible state). However, if the server is
sending the "hostkeys" global request, that usually happens at the same
time as the client is opening its first channel. Therefore, client-side
crash becomes very likely.

> Is RFC 8308 the right reference for this option? Are there
> any implementations of this so far other than from Bitvise?

Yes, RFC 8308 is the authoritative spec for the "delay-compression"
extension. Bitvise SSH Server and Client can be considered reference
implementations. As of right now I haven't been informed of others.

> Also, I’m not familiar with the PuTTY issue around needing a
> second key exchange. Is that documented somewhere?

The race condition is inherent in the [email-masked] algorithm. PuTTY
identified this issue and addressed it with a strategy based on key
re-exchange. Bitvise SSH Client also uses the key re-exchange strategy if
the server advertises [email-masked] but not the "delay-compression"
extension.

PuTTY project source on this issue:

https://www.chiark.greenend.org.uk/~sgtatham/putty/wishlist/zlib-openssh.html

denis

On Sun, May 3, 2020 at 1:14 AM Ron Frederick <[email-masked]> wrote:

> Yes, the idea of "global-requests-ok" is not to make the client do
> anything different. It's to reassure the server that the client won't
> violate the spec if the server sends a global request.
>
> The main motivating usage case is so the server can feel at liberty to
> send the global request "[email-masked]" after user
> authentication:
>
> https://cvsweb.openbsd.org/src/usr.bin/ssh/PROTOCOL?annotate=HEAD
>
> This is essential functionality which allows clients and servers to change
> host keys without requiring administrative intervention at each client
> installation.
>
> Without "global-requests-ok", the server needs a database of client
> version strings to which it's safe to send the "hostkeys" global request.
> Otherwise, it can't send it because a bunch of half-baked clients (I'm
> counting 7 in our list) are borked and can't handle receiving a global
> request. Or they can handle it usually, but not when they're expecting a
> response to a channel open.
>
>
> Yeah, understood. So far, I haven’t implemented “hostkeys" support in
> AsyncSSH. I may do that at some point, but I’ll probably just do the new
> host key validation in AsyncSSH but leave it up to the application code to
> decide if it actually wants to update the list of known_hosts or not (and
> if so, where). Right now, AsyncSSH will read from ~/.ssh/known_hosts by
> default if you don’t specify another source, but it never actually modifies
> that file. In fact, most of the ways of specifying known_hosts in AsyncSSH
> are focused on providing the set of hosts dynamically, or even allowing
> applications to provide their own logic to determine whether a specific
> host key should be trusted or not for a given host/port, so I’d want to do
> the same thing with reporting new host keys a server advertises.
>
> Since I don’t have this support yet, I haven’t needed to worry about which
> clients can properly handle global requests or not. I do use global
> requests in both directions as part of the implementation of "
> [email-masked]”, but so far I haven’t run into any issues with
> that. That’s disabled by default, though, so perhaps people are only
> enabling with peers that don’t have issues with global requests.
>
>
> Meanwhile I've received more info about the implementation that
> disconnects on receiving EXT_INFO from the client. The bug is unfortunately
> in a widely used Java SSH library. The information I received suggests it's
> going to be fixed, but I'm afraid there are already a fair number of
> deployed servers that have the issue.
>
>
> Interesting. Does it fail on ANY global request being sent from a server,
> or is it only at certain points that it doesn’t handle these messages
> correctly?
>
>
> While I'm wishing things - I would also like the "delay-compression"
> extension to be more widely supported because it's the only
> delayed-compression mechanism that (1) does not have a race condition by
> design ([email-masked]), and (2) does not require a second key exchange
> after user authentication (PuTTY mitigation).
>
>
> Is RFC 8308 the right reference for this option? Are there any
> implementations of this so far other than from Bitvise?
>