Subject: Practical forwarding problem

From: der Mouse <[email-masked]>
Date: Thu, 22 Sep 2005 20:47:55 -0400 (EDT)

I've finally got around to cutting over to my own ssh implementation on
my home machines.  I've run into a problem.  I know where it's coming
from but I'm not sure what the best tack to take on dealing with it is.

I'm writing the list because I'd be interested to hear if anyone else
has dealt with the same problem, and if so how, and also interested in
any thoughts anyone may have on it.

What's happening is:

- User authentication succeeds.
- Client opens a "session" channel (call it A).
- Server confirms the open.
- Client requests agent and X11 forwarding on A.
- Server accepts forwardings.
- Client sends an "exec" request on A.  The requested command is an X
   terminal emulator.
- Terminal emulator starts (opening an X connection) and backgrounds
   itself (which looks like dying, to the server).
- Server notices command death and sends EOF and CLOSE on A.
- Client sends CLOSE on A, acking the server's CLOSE.
- Server tears down forwardings for A, 'cause the channel's closed.
- Something running under the terminal emulator does something which
   wants agent access, or which wants a new X connection, but the
   forwarding is now gone!

Obviously, something here needs to change.  But what?  Everything
leading up to the problem looks reasonable, but it sure adds up to
"broken" in practice.  The most dubious point I see is the terminal
emulator backgrounding itself - but that worked fine with the old ssh1
implementation, and it doesn't seem totally unreasonable to expect it
to work still.

Thoughts, anyone?

/~\ The ASCII				der Mouse
\ / Ribbon Campaign
 X  Against HTML	       [email-masked]
/ \ Email!	     7D C8 61 52 5D E7 2D 39  4E F1 31 3E E8 B3 27 4B

---

From: Joseph Galbraith <[email-masked]>
Date: Thu, 22 Sep 2005 19:30:59 -0600

Only two possibilities really occur to me...

> - Server notices command death and sends EOF and CLOSE on A.

The server could notice that their are still child processes alive
and send EOF but not CLOSE.  Not great, since it might be that the
user intended these child processes to run in the background after
logout...

> - Server tears down forwardings for A, 'cause the channel's closed.

The Server could 'associate' the forwarding with the connection
instead of the channel... and not tear it down until the connection
dies.

In this model, a second agent-forwarding request on a second session
channel would not result in a second unix domain socket being created,
but would simply result in th environment variable being set to point
to the 'connection' forwarding.

The same for x11...

I'm not sure this is a good idea either... but it might be better than
the first one.

--

In the end, I think it comes down to the fact that agent and x11
forwarding really should be connection level operations... but
they are requested over a session channel because they require
setting environment for a given session.

I think it would have been better to make them connection
level requests, and then, maybe, have the session level
'forward-x11' request imply that a connection level forwarding
had already been set up and this was a request to make it
available to this particular session.

Waaaaaaay to late now, though...

Thanks,

Joseph

---

From: Frank Cusack <[email-masked]>
Date: Thu, 22 Sep 2005 21:35:18 -0700

> implementation, and it doesn't seem totally unreasonable to expect it

This is a known problem, for openssh at least.  You shouldn't send
EOF and CLOSE (are these actual protocol messages?) on command termination,
you should only send it on EOF from the read fd connected to the command's
stdout.

The problem with that is you end up with "hung" sessions if you start
a background process that doesn't close it's stdout (e.g. attempting
to run a process as a daemon that really is meant to run in the foreground).
So a compromise is that you note process death but wait until your select()
or poll() loop indicates that there is no data ready.

For some reason, openssh on BSD and openssh on Linux act differently here,
due to kernel differences.  It's been so long that I don't remember the
details.

Now, if you implemented the compromise method, this wouldn't solve your
problem, because an idle xterm isn't going to have data ready.

But that's all a red herring, because really you should notice that the
forwarded session is still open and delay closing the session until
the forwarding has gone bye-bye.  At least, that's what openssh does.

Even if I'm off track here, I suggest sending a note to the openssh
list.  It sounds off topic for this list.

-frank

---

From: der Mouse <[email-masked]>
Date: Fri, 23 Sep 2005 01:03:29 -0400 (EDT)

Replying to three messages at once, here.

>> [issue with forwardings being torn down before they're wanted]

[Joseph Galbraith <[email-masked]>]

> and send EOF but not CLOSE.

This is approximately impossible on at least some of my target OSes; if
A forks B and B forks C and B then dies, A has no way (well, no easy or
clean way) to find out that B forked C at all, much less that C still
exists.

> Not great, since it might be that the user intended these child
> processes to run in the background after logout...

There's that, too.

> instead of the channel... and not tear it down until the connection

That breaks connection sharing.  What I considered is allowing the
active forwardings to keep the session channel alive when it would
otherwise be closed, but then there's the problem of telling when you
*do* want to shut it down.  (But see David Terrell's idea, below.)

> channel would not result in a second unix domain socket being
> created, but would simply result in th environment variable being set
> to point to the 'connection' forwarding.

That sounds good, except it breaks if the two agent forwarding requests
are trying to provide forwarded access to different agent
instantiations.  (Mutatis mutandis for X, of course.)

> forwarding really should be connection level operations...

Perhaps, but that point of view is fundamentally incompatible with
connection sharing (which is *very* desirable for someone who, like me,
is on old half-lung hardware that can quite literally take minutes to
do kex and authentication).  Or at least with connection-sharing for
decoupled sessions.

[David Terrell <[email-masked]>]

> Option 1) Keep the terminal open until you're done

This is what I've started doing.  I didn't like it at all at first, and
still don't really like it that much, but I've made a mental shift to
seeing the old way's working as an accident of that implementation
rather than a "how it's supposed to work", which makes it easier to
accept the difference when I switch.

> Option 2) Keep open all forwardings as long as there's
> 	a) a forwarded session
> 	b) a forwarded x session

That is a sensible suggestion.  I may try it someday.

The real problem here is that I want to hand a capability to a process,
to pass on to descendants, and I want to know as soon as nobody who has
it is still alive any longer.  The only thing I can think of which fits
this in stock Unix is a file descriptor.  I could do that for agent
access, but I don't think I could for X access, not without hacking up
Xlib as well - and even for agent access, it's hard to get the
descriptor past things like shells which (not unreasonably) go to some
lengths to close all file descriptors they don't understand.

[Frank Cusack <[email-masked]>]

> EOF and CLOSE (are these actual protocol messages?)

Yes - SSH_MSG_CHANNEL_EOF and SSH_MSG_CHANNEL_CLOSE, to give them their
full names.

> on command termination, you should only send it on EOF from the read
> fd connected to the command's stdout.

Perhaps, but...

> a background process that doesn't close [its] stdout [...]

...I've run into that kind of wedgitude in the past, so I'm reluctant
to do that in that form.

> So a compromise is that you note process death but wait until your
> select() or poll() loop indicates that there is no data ready.

That helps with races between process exit and output flushing, but not
with a situation like this one, where the process quite likely will
_never_ produce any output.

> But that's all a red herring, because really you should notice that
> the forwarded session is still open and delay closing the session
> until the forwarding has gone bye-bye.  At least, that's what openssh
> does.

Not according to connect-25:

   X11 connection forwarding should stop when the session channel is
   closed.  However, already opened forwardings should not be
   automatically closed when the session channel is closed.

To me, that quite clearly says that I *should* shut down the forwarding
when the session channel closes.

> list.  It sounds off topic for this list.

Really?  OpenSSH is not involved in any way in the cases in which I ran
into this.

Anyway, thanks for all the thoughts.  I'll let them (and any more that
may show up) rattle around for a day or two and see if I can't settle
on something.

/~\ The ASCII				der Mouse
\ / Ribbon Campaign

---

From: Frank Cusack <[email-masked]>
Date: Thu, 22 Sep 2005 23:27:44 -0700

> when the session channel closes.

I don't read that.  What I read is that new X11 requests should no longer
be accepted, but that existing (_already opened_) forwardings should live
on.  The point of the paragraph is to note that just because an already
opened [X11] forwarding is left open, doesn't mean new X11 requests must
be honored.

Interesting that the text does not capitalize SHOULD and SHOULD NOT.

> Really?  OpenSSH is not involved in any way in the cases in which I ran

No, but they deal with this case.  My suggestion was just that you discuss
it with an implementor that has addressed it.  But now that I have a
clearer idea of what you're asking, maybe this list is more appropriate.

-frank

---

From: der Mouse <[email-masked]>
Date: Fri, 23 Sep 2005 02:31:09 -0400 (EDT)

>> To me, that quite clearly says that I *should* shut down the
>> forwarding when the session channel closes.

> I don't read that.  What I read is that new X11 requests should no
> longer be accepted, but that existing (_already opened_) forwardings
> should live on.

I suspect we may be in furious agreement here, just using different
words to describe it.  In particular, a "forwarding" may be different
things to you and to me.

By "new X11 requests" do you mean "x11-req" CHANNEL_REQUESTs, do you
mean new X connections on the server host (and the "x11" CHANNEL_OPENs
that result from them), or what?

By "existing...forwardings" do you mean existing channels opened with
type "x11", do you mean the listen for new X connections that's set up
on the server, or what?

My reading is that the "X11 connection forwarding should stop" means
that the server side should stop listening for new X connections and
"already opened forwardings should not be automatically closed" is
referring to existing X connections, and their associated "x11"
channels.

And when I equate that to "shut down the forwarding when the session
channel closes", I'm talking about the server stopping listening for
new X connections.

>> Really?  OpenSSH is not involved in any way in the cases in which I
>> ran into this.
> No, but they deal with this case.  My suggestion was just that you
> discuss it with an implementor that has addressed it.

Ah, I think I see what you mean.  Not so much "take it to openssh" as
"take it to a implementation list rather than protocol design list"?
Perhaps a parallel list for implementation issues would be appropriate.

> But now that I have a clearer idea of what you're asking, maybe this
> list is more appropriate.

Well, perhaps.  It *is* an implementation issue, not something that has
anything to do with the protocol per se - but the spec is intended to
guide implementations, and if the spec ends up compelling something
that cripples implementations, it needs fixing.  (In this case, I now
think it's not that bad, but it looked at first as though it might be.)

/~\ The ASCII				der Mouse
\ / Ribbon Campaign

---

From: Damien Miller <[email-masked]>
Date: Fri, 23 Sep 2005 17:35:26 +1000 (EST)

---

From: Frank Cusack <[email-masked]>
Date: Fri, 23 Sep 2005 02:32:22 -0700

On September 23, 2005 2:31:09 AM -0400 der Mouse <[email-masked]> wrote:
> I suspect we may be in furious agreement here,

We are, and I misread your original message as saying that the xterm
started in the background is no longer functional.  If you start a
command in that xterm which requires a new X connection, it should
fail, from my reading of connect-25.

However, I realize now that I don't quite understand the difference
between a session and a channel, so I'll bow out, hopefully gracefully.

Interestingly, this has just come up as an openssh bug,
<http://bugzilla.mindrot.org/show_bug.cgi?id=1086>.

-frank

---

From: [email-masked] (Niels Möller)
Date: 23 Sep 2005 13:34:41 +0200

> any thoughts anyone may have on it.

I have a couple of comments:

You say it's dubious to have the terminal emulator background itself.
I agree. Don't do that; if you want to have your local shell prompt
back, you should background the *local* ssh client, not the remote
terminal emulator. Some clients have a command line option "background
process after initial handshake and user authentication is complete".

> - Server notices command death and sends EOF and CLOSE on A.

The server should merely generate a CHANNEL_REQUEST "exit-status" on
process death, not CHANNEL_EOF or CHANNEL_CLOSE here. (The correct
close logic is somewhat hairy, but it's not really central to what you
are asking about).

> - Server tears down forwardings for A, 'cause the channel's closed.

Whenever the session channel is closed, it makes sense to tear down
the forwardings (i.e. let existing forwarded channels live on, but
don't allow opening of any new ones, and close any related listening
sockets, etc).

>    wants agent access, or which wants a new X connection, but the

If the session channel is dead, I think this is the correct behaviour.
Even if the server keeps its side of the forwarding open, it would
make sense for the client side to refuse to open new forwarded
channels for a closed sesssion. (And here we get into the connection
sharing issues; the client has some difficulty in handling the request
properly, since it's hard to identify a CHANNEL_OPEN request with the
session for which the corresponding forwarding was set up, in
particular if that session is dead, closed and deallocated.)

Regards,
/Niels