Subject: Protocol ambiguity: want_reply vs CHANNEL_CLOSE

From: Simon Tatham <[email-masked]>
Date: Fri, 04 Apr 2014 10:58:47 +0100

I think this list still reaches a reasonable number of SSH
implementors. I'd like to see if we can get a consensus on the
following problem.

Suppose two SSH implementations have a channel open between them.
Party A sends an SSH_MSG_CHANNEL_REQUEST of some kind, with the
want_reply flag set. Party B, meanwhile, independently decides to
initiate shutdown of the channel, and sends SSH_MSG_CHANNEL_CLOSE.
These two messages cross in the network, so that B receives the
channel request after it's already sent a close message for that
channel.

Should B send a reply?

I think you can argue it either way from the spec. RFC 4254 section
5.4 says that channel requests must be replied to if they have
want_reply set, and doesn't make any exception for channels in the
middle of closing, which supports the answer 'yes'. Then again,
section 5.3 says that after a party has both sent and received CLOSE,
the channel is defunct and its number can be reused, which supports
the answer 'no' (because, if B does reply, what if that reply crosses
with A's responding close message? Then B's reply would arrive at A
after A has both sent and received CLOSE).

So I think this is a genuine ambiguity in the RFC. Moreover, I've
observed SSH servers disagreeing on whether to reply, which makes it
fiddly to implement a policy at the request-sending end that handles
both kinds of server.

Would anyone like to state an opinion on what ought to happen about
this? I can think of two reasonably sensible options:

 (a) Issue a clarifying RFC that resolves the ambiguity by designating
     one or other behaviour as correct, and consider implementations
     doing the other thing to have a bug requiring a fix.

 (b) Issue an RFC specifying a protocol extension that allows an
     implementation to advertise which interpretation it will follow
     (since either behaviour is easy to cope with if you know which
     one you're talking to). I suppose we'd still have to decide on
     the default answer in the absence of such an advertisement.

Cheers,
Simon
-- 
Simon Tatham         "The distinction between the enlightened and the
<[email-masked]>    terminally confused is only apparent to the latter."

---

From: Matt Johnston <[email-masked]>
Date: Fri, 4 Apr 2014 21:09:45 +0800

> channel request after it's already sent a close message for that

I'd say no. It's hard to imagine a situation where the reply
would not be SSH_MSG_REQUEST_FAILURE, and a requestor
wouldn't be able to do much with a "success" reply in any
case. In addition it adds an another state for
implementations to handle, "both sides closed but waiting
for a reply".

Given there are millions of deployed servers that won't
change, would it be best to just publish a RFC and keep
using a compatibility table? Out of interest has PuTTY
encountered this problem in the wild?

FWIW Dropbear ignores requests after having sent
SSH_MSG_CHANNEL_CLOSE, though that was more from omission
of the wantreply case. All SSH_MSG_CHANNEL_REQUEST messages
have wantreply=0, so no replies are expected.

Cheers,
Matt

---

From: Simon Tatham <[email-masked]>
Date: Fri, 04 Apr 2014 16:00:59 +0100

Matt Johnston <[email-masked]> wrote:

> Given there are millions of deployed servers that won't change, would
> it be best to just publish a RFC and keep using a compatibility table?

In other words, designate one answer as 'right' and recommended for
new servers, but also keep a list of implementations known to do the
other thing? I suppose that's not too onerous; it's a bit out of the
ordinary as bug workarounds go, but then a fundamental protocol issue
that's been around forever _is_ a bit out of the ordinary.

> Out of interest has PuTTY encountered this problem in the wild?

Yes. We use a channel request with want_reply set as a means of
measuring round-trip time so as to tune our window size, which means
that if a server closes a channel at all it's quite common for us to
have just sent a channel request at the point it does so.

And yes, we have encountered at least one server with each policy!

Cheers,
Simon
-- 
Simon Tatham         "Every person has a thinking part that wonders what
<[email-masked]>    the part that isn't thinking isn't thinking about."

---

From: Simon Tatham <[email-masked]>
Date: Fri, 04 Apr 2014 16:29:31 +0100

"denis bider \(Bitvise\)" <[email-masked]> wrote:

> The behavior that accommodates all request responders is as follows:
[...]
> - Do not reuse channel numbers, so that you can reliably ignore channel 
> request responses received after CLOSE.

The trouble with that is that it's _horrible_! The intention of the
SSH protocol design was always that channel numbers should be
reusable, so that SSH sessions do not have a designed-in lifetime
limit.

2^32 channel opens and closes is certainly a lot, but it isn't beyond
the bounds of belief in situations of (for example) heavy WWW use over
SOCKS forwarding in a long-lived connection.

It may be necessary to use this as a kludgy workaround for some
particular server, if we really cannot determine from its version
string which policy it follows, but I cannot bring myself to believe
it's the sensible policy to apply in all situations.

Cheers,
Simon
-- 
Simon Tatham         What do we want?        ROT13!
<[email-masked]>   When do we want it?     ABJ!

---

From: "denis bider \(Bitvise\)" <[email-masked]>
Date: Fri, 4 Apr 2014 09:11:15 -0600

> Out of interest has PuTTY encountered this problem in the wild?

Yes, with our server (Bitvise).

The way I see this issue is that in the described situation, the request 
sender can implement behavior that accommodates all responders, but the 
request responder cannot implement behavior that accommodates all request 
senders.

- Do not wait for channel request responses after receiving CLOSE.

On the responder's end, however, there are these choices:

- If a channel request arrives after we sent CLOSE, but before we received 
remote CLOSE, ignore it. This will work for request senders which don't wait 
for request responses after CLOSE, but it will hurt request senders that do.

- Alternately, if a channel request arrives after we sent CLOSE, but before 
we received remote CLOSE, reply to it. This will work for request senders 
which wait for request responses after CLOSE, but it creates a possible race 
condition for request senders which do not:

  (a) The request sender might have already sent CLOSE that we haven't 
received yet, might have freed channel state, and might abort the session 
due to unexpected channel number in request response.

  (b) Alternately, the request sender might have already sent CLOSE that we 
haven't received yet, and immediately after, respond to our CHANNEL_OPEN 
that we might have just sent. The request sender might then interpret our 
response as arriving on that new channel, rather than the previous one being 
closed.

With millions of SSH implementations out there, we are no longer in a 
position to mandate behavior that makes implementations incompatible with 
one another. I think we ought to advise in favor of the most compatible 
strategy, which is:

- Do not reuse channel numbers.

- Do not send responses after sending CLOSE.

- Do not wait for responses after receiving CLOSE.

denis

-----Original Message----- 
From: Matt Johnston
Sent: Friday, April 4, 2014 07:09
To: Simon Tatham
Cc: [email-masked]
Subject: Re: Protocol ambiguity: want_reply vs CHANNEL_CLOSE

Cheers,
Matt

---

From: "denis bider \(Bitvise\)" <[email-masked]>
Date: Fri, 4 Apr 2014 09:38:30 -0600

You don't have to avoid reusing ALL channel numbers, just recent ones.

What our implementation does is:

- Increment channel number up to 2^32, then wrap around.

- If the channel number we're trying to use is already in use, try next one.

You could also conceivably do something else, like:

- Reuse channel numbers, but

- Not channel numbers that were used within the last X minutes of operation.

denis

-----Original Message----- 
From: Simon Tatham
Sent: Friday, April 4, 2014 09:29
To: denis bider (Bitvise)
Cc: Matt Johnston ; [email-masked]
Subject: Re: Protocol ambiguity: want_reply vs CHANNEL_CLOSE

[...]

Cheers,
Simon

---

From: [email-masked] (Niels Möller)
Date: Sun, 06 Apr 2014 14:45:27 +0200

> channel request after it's already sent a close message for that

I'd say no. I think it is pretty clear that the intention is that the
CLOSE message is the final one. So that the receiver of the CLOSE close
message can respond with a CLOSE, unless it has already sent one, and
then forget *all* state of the channel.

(Maybe one also have the option to sending additional messages before
responding with a matching CLOSE. I don't know if any implementations do
that. I think receiving a CLOSE means that the other end has lost all
interest in the channel, and then it doesn't make any sense to send
anything but a matching CLOSE).

>      doing the other thing to have a bug requiring a fix.

I think it would be useful to sort out what problems that bug causes,
and if there are any reasonable workarounds. I think there are a couple
of cases:

1. We receive an unexpected response (which the sender intended for a
   now dead channel). Treated an a protocol error, and handled by
   closing the connection. Here's one has the option of ignoring
   unexpected responses, possibly depending on the other side's version
   string.

2. We send a CHANNEL_REQUEST for a channel and get two replies (where
   the first was intended for a no longer alive channel). I imagine this
   could lead to more subtle problems. This is the case that can be
   eliminated by following the suggestion of not reusing channel
   numbers.

3. The implementation at the other end somehow expects a response after
   CLOSE, and not sending that breaks it's channel close handshake
   logic, leaving a channel alive for ever. I hope this problem doesn't
   require any new workarounds: The other end is typically untrusted,
   and we shouldn't let deviations like this cause us any big problems.

Can you give us some more details on the problems you have encountered?
Maybe there are simple workarounds for 95% of the problems, maybe
there isn't.

Regards,
/Niels

-- 
Niels Möller. PGP-encrypted email is preferred. Keyid C0B98E26.
Internet email is subject to wholesale government surveillance.

---

From: Simon Tatham <[email-masked]>
Date: Mon, 07 Apr 2014 14:45:29 +0100

> Can you give us some more details on the problems you have encountered?

Background: as I mentioned in a previous message, PuTTY periodically
sends a bogus channel request "[email-masked]",
with want_reply set. There is no correct handling defined for this
request, so we expect CHANNEL_FAILURE from the server (though we know
of at least one server which amusingly sends CHANNEL_SUCCESS :-). We
use the timing of that failure message to gauge the round-trip time
and tune our window size. All problems I've seen with this race
condition have related to these winadj messages. (So the usual
workaround if a server is not playing nicely with PuTTY is to suppress
winadjes.)

The first problem we noticed was precisely your #1: PuTTY would crash
out with 'Received SSH2_MSG_CHANNEL_FAILURE for nonexistent channel
256' type messages, which turned out to be because the server was
sending us CHANNEL_FAILURE after CHANNEL_CLOSE whereas we were
expecting that CHANNEL_CLOSE meant the server would send no further
messages about that channel.

We observed this with several servers: my email archives mention
Dropbear 0.51 in 2008, OpenSSH 5.1p1 in 2010, and at least one or two
SSH servers in embedded devices for which I didn't get full details.
Dropbear promptly fixed the problem (changing the server to stop
sending post-close CHANNEL_FAILURE). OpenSSH did not, because Damien
Miller questioned my analysis of it as a server-side bug on the basis
that the spec was unclear (which indeed it is):

  https://bugzilla.mindrot.org/show_bug.cgi?id=1818

Since OpenSSH didn't make any changes, and since I'd forgotten the
previous Dropbear incident at that time, I implemented PuTTY's current
workaround, which is to defer _sending_ CHANNEL_CLOSE until it's seen
all outstanding CHANNEL_FAILURE messages, so that (from our POV) the
channel is still not completely closed at the point when those
failures arrive. Unfortunately, it apparently didn't occur to me that
that would break interoperation with servers that _don't_ send the
outstanding failure messages, because now PuTTY sits and waits for a
failure message that will never arrive and so users see hangs on the
client side.

So, clearly some kind of workaround is needed. But separately from the
question of stopgap workarounds while we all sort ourselves out, I'd
like to establish a consensus on what the right behaviour _is_, so
that SSH implementations can gradually converge on that, and so that
in any further situation where a client and a server don't work well
together, there's a document to point to which will make it clear
which one is wrong, so that we don't have 'no, _you_ fix it' deadlocks
between equally stubborn implementors.

Responses so far have all said that the consensus is to rule that
sending FAILURE after CLOSE is wrong, and that agrees with my original
belief (before I started finding out that not everyone agreed). The
only people I know who have so far _deliberately_ gone with the other
behaviour (in the sense of arguing against fixing it when it was
pointed out) are the OpenSSH maintainers. Are any of them reading
this?

Cheers,
Simon
-- 
Simon Tatham         "I thought I'd put my foot so far into my mouth I
<[email-masked]>    wouldn't be able to sit down without standing up."

---

From: "denis bider \(Bitvise\)" <[email-masked]>
Date: Mon, 7 Apr 2014 08:51:03 -0600

I really still think this is a simple issue for the request sender to solve.

After CLOSE is received, instead of keeping the channel alive indefinitely, 
keep it in a zombie state for up to X minutes. Don't make a channel in this 
zombie state keep a session open, if it would otherwise be closed. If you 
receive any outstanding responses during zombie state, this lets you 
recognize them and ignore them and not mix them up with any other channels 
that might have been opened in the meanwhile.

You don't need to know what OpenSSH implementors think because this strategy 
works regardless of their opinion. You don't need a spec or a community 
consensus to back this strategy because you don't have to defend this 
strategy against anyone; it works with all request responders.

It simply makes sense to implement.

-----Original Message----- 
From: Simon Tatham
Sent: Monday, April 7, 2014 07:45
To: Niels "Möller"
Cc: [email-masked]
Subject: Re: Protocol ambiguity: want_reply vs CHANNEL_CLOSE

Cheers,
Simon

---

From: [email-masked] (Niels Möller)
Date: Mon, 07 Apr 2014 22:38:59 +0200

Simon Tatham <[email-masked]> writes:

> of at least one server which amusingly sends CHANNEL_SUCCESS :-).

I think I've heard of keep-alive requests which work the same way.

Anyway, in the work-around category, does this have to be a channel
request, or could you use a global request instead? (I think I've heard
of servers disconnecting on unknown global requests, but that was some
years ago).

> OpenSSH did not, because Damien

The spec is a bit unclear, but I think your initial interpretation is
the one which makes the most sense. Consider the sequence (which I think
you mentioned in some comment):

  A sends CHANNEL_REQUEST (with want_reply set), followed by
  CHANNEL_CLOSE.

  B responds with a CHANNEL_CLOSE, followed by a CHANNEL_FAILURE.

At the time A receives the CHANNEL_CLOSE, section 5.3 is pretty clear
that the channel is dead, and that A may reuse the channel number. So in
this scenario the CHANNEL_FAILURE clearly violates the protocol spec.
Note that it's not really possible for A to know if B's CHANNEL_CLOSE
was a response to A's, or if it was sent while A's messages were still
in flight.

Now, the question is where to put the blame for this violation. I'd say
put the blame on B: After CHANNEL_CLOSE, no further messages relating to
the channel can be sent, period. The alternative is to say that it's A's
fault: A shouldn't send CHANNEL_CLOSE while it has a pending request. I
see two related problems with this alternative:

1. Say A sends a CHANNEL_REQUEST, which B never responds to. Maybe B is
   misbehaving, or maybe progress on the request simply depends on some
   process which is stuck or otherwise takes a very long time to
   complete. Then it will not be possible for A to tear down the
   channel; since it hasn't seen the response, it would be forbidden to
   send a CHANNEL_CLOSE. While I think the intention is that either
   party can initiate teardown by sending CHANNEL_CLOSE any time it
   likes.

2. It doesn't quite handle the case B generated the CHANNEL_CLOSE
   independently of A's actions. If B sends the CHANNEL_CLOSE while A's
   CHANNEL_REQUEST is in flight, the spec still requires A to respond
   with a CHANNEL_CLOSE. I think it's fairly clear that the intention is
   that that response should be prompt, with no long delay. On the other
   hand, there's no limit on how long it might take to respond to a
   CHANNEL_REQUEST.

I'm not sure how the update/errata process works. I think it would make
sense to amend section 5.3 to say something like:

  SSH_MSG_CHANNEL_CLOSE is the final message a party for a channel. If,
  at the time a party receives a SSH_MSG_CHANNEL_CLOSE, there are some
  SSH_MSG_CHANNEL_REQUEST that party has sent but not received any
  replies for, the SSH_MSG_CHANNEL_CLOSE message should be treated as an
  implicit SSH_MSG_CHANNEL_FAILURE for all outstanding requests

The lst piece is actually a bit subtle. If you have received a
CHANNEL_REQUEST, and started acting on it at the time you receive a
CHANNEL_CLOSE, and the request has any potential side effects, you would
need to somehow cancel it. As long as it is still possible that the
request might succeed, you can't send the required CHANNEL_CLOSE
response. In practice, I think a *short* delay to sort this out is
perfectly acceptable.

> The only people I know who have so far _deliberately_ gone with the
> other behaviour (in the sense of arguing against fixing it when it was

The way I read that bug log, at least they don't say they won't fix it
ever. So the low priority doesn't necessarily mean that they are
convinced that there is no openssh bug here.

Regards,
/Niels

---

From: Damien Miller <[email-masked]>
Date: Sun, 13 Apr 2014 21:30:52 +1000 (EST)

> channel request after it's already sent a close message for that

I agree the spec doesn't make this clear but IMO no. 'A' will receive
SSH_MSG_CHANNEL_CLOSE before it would have received any reply and
should, based on it, abandon it expectation that a reply will be
forthcoming.

>      doing the other thing to have a bug requiring a fix.

^^^ this.

>      the default answer in the absence of such an advertisement.

IMO no - it's overkill and not necessary.

-d