Subject: RE: [Simple] MSRP: Administrative shutdown of relays

From: [email-masked]
Date: Sun, 12 Oct 2003 14:28:12 +0300

> -----Original Message-----
> From: ext Ben Campbell [mailto:[email-masked]]
> Sent: Friday, October 10, 2003 11:47 PM
> To: Paul Kyzivat
> Cc: Khartabil Hisham (NMP-MSW/Helsinki); [email-masked];
> [email-masked]
> Subject: Re: [Simple] MSRP: Administrative shutdown of relays
> 
> 
> Paul Kyzivat wrote:
> 
> > 
> > 
> > Ben Campbell wrote:
> > 
> >>
> >> That is not far from the same thing, since if there was a 
> single idle 
> >> session, it will take up to 1 timeout for each endpoint to send a 
> >> request that the relay could respond to.
> >>
> >> But my point was, for Paul's scenario, it is not 
> neccessary to wait 
> >> for idle endpoints to get around to sending a request. You 
> just have 
> >> to wait long enough so that it is very unlikely that any 
> transactions 
> >> are still in play. A small number of minutes should be 
> plenty of time.
> > 
> > 
> > Hmm. How does waiting decrease the likelihood that messages 
> are still in 
> > play? Even though there have been no messages in the last N 
> minutes, one 
> > could just have been sent.
> > 
> > To be sure there are none, I think you must in general wait for the 
> > timeout interval if the stream is idle. But if the stream 
> is active, you 
> > can just start sending the magic error code to every 
> received message 
> > for awhile, with the understanding that a sending, upon 
> receiving this 
> > code, will no longer send messages on the stream. Then, 
> after enough 
> > time has passed for all messages in the pipeline to have 
> been received, 
> > then stream can be shutdown.
> 
> If the goal is to make the relay notifies all endpoints prior to 
> shutting down, then yes, you would have to wait for a full inactivity 
> timeout period. But that was not the scenario you subscribed. If the 
> relay begins sending 500 responses to any message at time T, 
> then it can 
> assume that there are no more outstanding transactions at 
> T+N. Where N 
> is the _transaction_ timout, rather than the inactivity 
> timeout. (Now, 
> we do not actually define the transaction timeout--it is currently 
> listed as some "reasonable period of time." Maybe we need to specify 
> that, but I am not sure this particular reason is enough to 
> force that 
> issue.)
> 
> So, at time T+N, the relay can reasonably assume that any transaction 
> that was already in progress when it started sending the 500s 
> has either 
> completed or timed out. I suppose there could be requests 
> that are still 
> sitting in the endpoints TCP send buffer and never get a 500, 
> but that 
> would be a pretty small percentages of messages overall.
> 
> > 
> > To speed up shutdown of an inactive stream I think either 
> the inactivity 
> > timeout needs to be reduced, or else there needs to be a special 
> > graceful close message that can be sent.
> 
> I gather that you are looking for a "perfectly" graceful 
> shutdown, that 
> guarantees that the endpoint will know for sure that a given 
> message has 
> been delivered or not delivered.  I accept that it may be polite 
> behavior to minimize messages in "limbo". But nothing in MSRP 
> guarantees 
> that a SEND transaction that times out without a response has not 
> actually been delivered. We can only guarantee that, if you get a 
> successful response back, the message _has_ been delivered.
> 
> So I think it is reasonable to suggest that a relay wait a couple of 
> minutes after it stops relaying requests before it actually 
> shuts down. 
> I don't think it is reasonable to ask it to wait 12 minutes, 
> and I don't 
> think that this particular issue provides sufficient motivation to 
> reduce the idle timer.
> 
> What do others think on this point?

If you don't wait the full idle timeout period, then I don't see the point in waiting a couple of mins before shutting down. It is better to just shut down immediately then.

As I said earlier, relay shuts down after the full timeout period, or until it know it has sent 4xx (or 5xx) responses to SEND requests sent by all the participants in all session. Whichever is reached first. In this way, you guarantee that either the clients have been notified of such shutdown on that the clients have timed out themselves anyway.

/Hisham

> >     Paul

_______________________________________________
Simple mailing list
[email-masked]
https://www1.ietf.org/mailman/listinfo/simple

---

From: Paul Kyzivat <[email-masked]>
Date: Mon, 13 Oct 2003 09:16:27 -0400

hisham - I agree.

	Paul

[email-masked] wrote:
> 
> If you don't wait the full idle timeout period,
 > then I don't see the point in waiting a couple of mins
 > before shutting down. It is better to just shut down
 > immediately then.
> 
> As I said earlier, relay shuts down after the full timeout
 > period, or until it know it has sent 4xx (or 5xx) responses
 > to SEND requests sent by all the participants in all session.
 > Whichever is reached first. In this way, you guarantee that
 > either the clients have been notified of such shutdown on
 > that the clients have timed out themselves anyway.

---

From: Paul Kyzivat <[email-masked]>
Date: Mon, 13 Oct 2003 09:16:27 -0400

hisham - I agree.

	Paul

---

From: Ben Campbell <[email-masked]>
Date: Mon, 13 Oct 2003 13:25:26 -0500

> If you don't wait the full idle timeout period, then I don't see the point in waiting a couple of mins before shutting down. It is better to just shut down immediately then.

OK, I think we are talking past each other, because it seems to me there 
is a big advantage in waiting a "couple of minutes". Again, the point of 
this waiting period does not have anything to do with notifying 
endpoints a relay is about to go down. The point is to stop accepting 
transactions some period before shutting down. This gives us a couple of 
scenarios:

1) A transaction arrives after we stop accepting new transactions.  We 
immediately reject it with a 5xx response. We do not have to wait for 
some downstream device to respond.

2) A transaction arrives before we stop accepting new transactions. 
Because we are waiting a reasonable period between the time we stopped 
accepting transactions, and the time we shutdown, we make it highly 
likely that the transaction will either complete, or be timed-out by the 
sender before we actually shut down.

So it seems to me that, if at time T we stop accepting transactions, and 
at time T + N, where N is approximately a normal transaction timeout, 
greatly reduces the opportunity for transactions going into the black 
hole because of a shutdown. (At least for transactions that were not 
going to get black-holed _anyway_.)

Sure, this is not perfect, as at the moment of shutdown, there may be 
indbound requests in transit at the transport layer. I think we can live 
with that.

> As I said earlier, relay shuts down after the full timeout period, or until it know it has sent 4xx (or 5xx) responses to SEND requests sent by all the participants in all session. Whichever is rea…

I don't think advance client knowledge that the relay is going to 
shutdown is a requirement. The client will discover this when the relay 
closes the TCP connection.

---

From: Ben Campbell <[email-masked]>
Date: Mon, 13 Oct 2003 13:25:26 -0500

---

From: Paul Kyzivat <[email-masked]>
Date: Mon, 13 Oct 2003 15:01:30 -0400

> indbound requests in transit at the transport layer. I think we can live 

Why is that a good thing to live with? It means that every time some big 
relay is shutdown for maintenance, some number of people will see one of 
their messages duplicated. Many of those people will file a bug report 
the first time this happens to them. All of them will consider the 
service buggy as a result.

Because of the bugginess, applications will either decide some other 
transport is better, or else will try to work something into the 
endpoint to detect the repeated message.

So I think we either need to prevent the duplicated messages (without 
introducing lost messages) or else we need to specify how the recovery 
works, including supression of duplicates.

	Paul

---

From: Paul Kyzivat <[email-masked]>
Date: Mon, 13 Oct 2003 15:01:30 -0400

Paul

---

From: Ben Campbell <[email-masked]>
Date: Mon, 13 Oct 2003 15:21:29 -0500

Paul Kyzivat wrote:

> 
> 
> Ben Campbell wrote:
> 
>>
>> OK, I think we are talking past each other, because it seems to me 
>> there is a big advantage in waiting a "couple of minutes". Again, the 
>> point of this waiting period does not have anything to do with 
>> notifying endpoints a relay is about to go down. The point is to stop 
>> accepting transactions some period before shutting down. This gives us 
>> a couple of scenarios:
>>
>> likely that the transaction will either complete, or be timed-out by 
>> the sender before we actually shut down.
>>
>> So it seems to me that, if at time T we stop accepting transactions, 
>> and at time T + N, where N is approximately a normal transaction 
>> timeout, greatly reduces the opportunity for transactions going into 
>> the black hole because of a shutdown. (At least for transactions that 
>> were not going to get black-holed _anyway_.)
>>
>> indbound requests in transit at the transport layer. I think we can 
>> live with that.
> 
> 

I did not say it was good. I said I think we can live with it. That is 
not the same thing.

It seems to me we have 4 choices:

1) Live with the potential for duplicate deliveries when a logical 
conversation transitions from one session to another in a short time 
period. Note that this can happen without relays being involved at all.

2) Require any hosting device that wishes to end a session wait either 
until connected devices send a request, so that we can send a response 
indicating that we are shutting down, or until an inactivity timeout occurs.

3) Allow relays to originate a message saying the session is going down.

4) Create a cross-session duplicate supression mechanism.

3) and 4) seem awfully heavy-weight for the value they create. 2) either 
puts a big burden on hosting device shutdown time, or requires us to 
significantly shorten the timer value--which, as has been discussed at 
length on this list, is also heavy-weight.

Of these choices, I prefer option 1.

---

From: Ben Campbell <[email-masked]>
Date: Mon, 13 Oct 2003 15:21:29 -0500

---

From: Paul Kyzivat <[email-masked]>
Date: Mon, 13 Oct 2003 17:18:17 -0400

> Of these choices, I prefer option 1.

I think you managed to summarize the choices quite well.

I prefer either 2, or 4. Of those, 2 is the simpler, and I would be ok 
with reducing the timer value to make it responsive enough. OTOH, 4 also 
works in the case of a server crashing ungracefully, which 2 won't cover.

I think 1 is a non-starter. We are all geeks, and can be tolerant of 
such misbehavior if we understand why it is happening. Others will not 
be so tolerant. They especially won't be tolerant if the message happens 
to be their 5gb movie.

Now that I think about it, the 5gb message may be a reason to prefer 4. 
(Or 3): I don't think 2 works for that case.

	Paul

---

From: Paul Kyzivat <[email-masked]>
Date: Mon, 13 Oct 2003 17:18:17 -0400

Paul

---

From: Robert Sparks <[email-masked]>
Date: Mon, 13 Oct 2003 18:11:24 -0500

(as non-chair).

I think the edge conditions we are talking about here are sufficiently
edge that we should document them and not take on the burden of trying
to prevent them. 

I prefer option 1.

RjS

---

From: Robert Sparks <[email-masked]>
Date: Mon, 13 Oct 2003 18:11:24 -0500

(as non-chair).

I prefer option 1.

RjS

---

From: Ben Campbell <[email-masked]>
Date: Mon, 13 Oct 2003 18:25:34 -0500

Ben Campbell wrote:

[...]

> period. Note that this can happen without relays being involved at all.

I did, btw, intend option 1 to have a long enough waiting period so that 
any transactions that were in play when we decided to shutdown have time 
to complete. I just failed to actually type it. :-)

---

From: Ben Campbell <[email-masked]>
Date: Mon, 13 Oct 2003 18:25:34 -0500

Ben Campbell wrote:

[...]

---

From: Paul Kyzivat <[email-masked]>
Date: Tue, 14 Oct 2003 10:57:33 -0400

I thought that Ben's timeout approach didn't work, but after looking at 
it more closely I have convinced myself that it does work after all. 
Here is my analysis:

With a single relay R, states of a given message are roughly:

1) message in transit from UAC to R
2) message being processed by R
3) message in transit from R to UAS
4) message being processed by UAS
5) response in transit from UAS to R
6) response being processed by R
7) response in transit from R to UAC

A UAC that sends a request only knows it is in one of those states, not 
which one. If the link to the UAC fails while a request is outstanding, 
the UAC will need to retransmit it on a new connection to ensure it gets 
through. That will be no problem if the message is in states 1-3, but it 
will result in message duplication if the message was in states 4-7. The 
trick is to ensure no messages are in states 4-7.

Ben proposed a shutdown strategy where R begins returning an error 
response for each message arriving at R, while continuing to relay 
previously received messages and both previously and newly arrived 
responses. Then, after a waiting period of time TW in this mode, it 
simply drops all of its connections.

Using this strategy, if the times for each state are bounded (T1...T7), 
and TW > (T2 + ...  + T7), then any remaining messages will be in state 
1. R can't do anything to prevent messages being in state 1, but it 
doesn't matter because when the UAC detects the error from shutdown, its 
recovery action of retransmitting them will be correct and will not 
result in duplication.

This supposes that the durations of the different states are reasonably 
bounded. This assumtion may be violated for large messages. But perhaps 
they aren't really a problem. For a large message, if R has not yet 
transmitted all of it on to UAS, it can simply cease transmitting more 
of it, and send an error response back to the UAC. The UAS will 
eventually realize there is an error when the connection to R goes down. 
As long as it doesn't present the partial message things will be ok.

This leaves a case where the message is large and all of it has been 
transmitted by R. If the latency between R sending the last byte of the 
message and receiving the response is not reasonably bounded then there 
is still a problem. This doesn't seem like a likely case.

Large responses would also screw things up, but I think we have defined 
things so that this is never the case.

The two relay case is more complex, but it feels like the results will 
be the same.

If nobody can shoot any holes in the above, I am prepared to agree with 
you that choice 1 is acceptable for relay shutdown, so that choices 2 & 
3 have no particular advantage over it.

There could still be value to choice 4. It would eliminate any need for 
waiting and sending errors during shutdown of a relay. In addition, it 
would also work for failover when a relay crashes. But I'm not sure I 
want this discussion to go there right now.

	Paul

---

From: Paul Kyzivat <[email-masked]>
Date: Tue, 14 Oct 2003 10:57:33 -0400

Paul

---

From: Ben Campbell <[email-masked]>
Date: Tue, 14 Oct 2003 10:37:48 -0500

> trick is to ensure no messages are in states 4-7.

Ah, you have just convinced me that the one problem I was worried about 
(i.e. state 1) is not really a problem. That is, requests that end in 
state 1 are known to have not been delivered, and resending them will 
not result in a duplicate. So I think that, rather than characterizing 
this approach as a partial solution, we can consider it a complete 
solution, pending correct selection of Tw.

> is still a problem. This doesn't seem like a likely case.

Yes. This brings us back to the concept of MSRP transaction timeouts. 
The draft currently says the sender should wait some "reasonable" period 
of time after sending the last byte of the message giving up on the 
transaction. A "reasonable" time should take the latencies you mention 
into account (perehaps for 2 relays.)

Previously, it was not important that all devices have the same 
definition of "reasonable". But it seems like the period the relay 
should wait before closing connections is the same as the period a 
client should wait before giving up on a transaction. So for this 
approach to work well, we may need to actually standardise that value.

> things so that this is never the case.

I certainly hope so.

> The two relay case is more complex, but it feels like the results will 

I think the 2 relay case adds more states, and possibly indicates a 
larger value for Tw. But if you define state 1 for any relay as meaning 
the request is somewhere between the client and the relay (possibly 
being processed by another intervening relay.), the model still holds up.

[...]

---

From: Ben Campbell <[email-masked]>
Date: Tue, 14 Oct 2003 10:37:48 -0500

I certainly hope so.

[...]