Subject: [Simple] MSRP congestion control issue when using relays

From: Ruud Klaver <[email-masked]>
Date: Tue, 11 Dec 2007 10:43:29 +0100

Hi,

At the risk of flogging a dead horse, I would like to re-raise an  
issue that was very kindly brought to my attention by Rémi Denis- 
Courmont.

In my attempts to implement a MSRP relay I am running into the  
problem of end-to-end congestion control. Congestion on an outgoing  
link will cause increasing buffer usage on a relay. This relay could  
do one of two things:
(a) Start dropping MSRP chunks of the congested session.
(b) Let the kernelspace TCP buffers fill up and have the TCP receiver  
window mechanism take care of quenching the incoming TCP connection.

In the case of multiplexing several MSRP sessions over the incoming  
TCP connection (b) would cause blocking in possibly uncongested  
sessions, while (a) clearly requires emulating some TCP-ish end-to- 
end congestion control.

In my view, re-solving a problem TCP has already dealt with on the  
MSRP level is not desired. I tried backreading some discussion about  
this in the archives of this mailinglist, and it seems that after  
some discussion in April of 2003 there was a consensus that session  
multiplexing should not be performed when using relays, i.e. each  
session should have its own TCP connection so that congestion control  
can be dealt with in the transport layer. To me this seems to be the  
best solution, although this does mean some incurred overhead of TLS  
negotiation for each session.

My question is, did I interpret this consensus correctly from the  
discussion and is it still valid? If it is, I could find no clear  
indication about this in the MSRP relay RFC, although maybe it is  
there and I did not spot it.

Also, I am aware that I may be confusing the concepts of congestion  
control and flow control, flow control may be a more appropriate term  
in my previous statements.

Ruud Klaver

_______________________________________________
Simple mailing list
[email-masked]
https://www1.ietf.org/mailman/listinfo/simple

---

From: "Sanjay Sinha (sanjsinh)" <[email-masked]>
Date: Tue, 11 Dec 2007 23:19:32 -0500

Inline ..

>-----Original Message-----
>From: Ruud Klaver [mailto:[email-masked]] 
>Sent: Tuesday, December 11, 2007 4:43 AM
>To: [email-masked]
>Subject: [Simple] MSRP congestion control issue when using relays
>
>Hi,
>
>At the risk of flogging a dead horse, I would like to re-raise 
>an issue that was very kindly brought to my attention by Rémi 
>Denis- Courmont.
>
>problem of end-to-end congestion control. Congestion on an 
>outgoing link will cause increasing buffer usage on a relay. 
>This relay could do one of two things:
>(b) Let the kernelspace TCP buffers fill up and have the TCP 
>receiver window mechanism take care of quenching the incoming 
>TCP connection.
>
>In the case of multiplexing several MSRP sessions over the 
>incoming TCP connection (b) would cause blocking in possibly 
>uncongested sessions, while (a) clearly requires emulating 
>some TCP-ish end-to- end congestion control.
>
>In my view, re-solving a problem TCP has already dealt with on 
>the MSRP level is not desired. I tried backreading some 
>discussion about this in the archives of this mailinglist, and 
>it seems that after some discussion in April of 2003 there was 
>a consensus that session multiplexing should not be performed 
>when using relays, i.e. each session should have its own TCP 
>connection so that congestion control can be dealt with in the 
>transport layer.

MSRP RFC (4975) mentions that a sender should not create another connection if one already exists to the host,port and msrp uri schema, rather reuse the existing connection.

Sanjay.

>To me this seems to be the best solution, 
>although this does mean some incurred overhead of TLS 
>My question is, did I interpret this consensus correctly from 
>the discussion and is it still valid? If it is, I could find 
>no clear indication about this in the MSRP relay RFC, although 
>maybe it is there and I did not spot it.
>
>Also, I am aware that I may be confusing the concepts of 
>congestion control and flow control, flow control may be a 
>more appropriate term in my previous statements.
>
>Ruud Klaver
>

---

From: Rémi Denis-Courmont <[email-masked]>
Date: Wed, 12 Dec 2007 09:37:05 +0200

Le Wednesday 12 December 2007 06:19:32 ext Sanjay Sinha (sanjsinh), vous avez 
écrit :
> >the MSRP level is not desired.

Agreed completely.

> >I tried backreading some discussion about this in the archives
> >of this mailinglist, and it seems that after some discussion in
> >April of 2003 there was a consensus that session multiplexing
> >should not be performed when using relays, i.e. each session
> >should have its own TCP connection so that congestion control
> >can be dealt with in the transport layer.
>
> MSRP RFC (4975) mentions that a sender should not create another connection
> if one already exists to the host,port and msrp uri schema, rather reuse
> the existing connection.

And you propose what instead... ?

-- 
Rémi Denis-Courmont

---

From: Ruud Klaver <[email-masked]>
Date: Wed, 12 Dec 2007 10:20:04 +0100

Hi,

On Dec 12, 2007, at 5:19 AM, Sanjay Sinha (sanjsinh) wrote:

> MSRP RFC (4975) mentions that a sender should not create another  
> connection if one already exists to the host,port and msrp uri  
> schema, rather reuse the existing connection.
>
> Sanjay.

This is exactly the statement that confused me in the first place. My  
argument is that this works fine in a situation where relays are not  
used (and I suppose RFC4975 only discusses this case as relays are  
split off into RFC4976), but when using relays one encounters the  
problem I described.

Ruud Klaver

---

From: "Sanjay Sinha (sanjsinh)" <[email-masked]>
Date: Wed, 12 Dec 2007 08:10:20 -0500

Inline ... 

>From: Rémi Denis-Courmont [mailto:[email-masked]] 
>Sent: Wednesday, December 12, 2007 2:37 AM
>To: [email-masked]
>Cc: Sanjay Sinha (sanjsinh)
>Subject: Re: [Simple] MSRP congestion control issue when using relays
>
>Le Wednesday 12 December 2007 06:19:32 ext Sanjay Sinha 
>(sanjsinh), vous avez écrit :
>> >MSRP level is not desired.
>
>Agreed completely.
>
>> >I tried backreading some discussion about this in the archives of 
>> >this mailinglist, and it seems that after some discussion 
>in April of 
>> >2003 there was a consensus that session multiplexing should not be 
>> >performed when using relays, i.e. each session should have its own 
>> >TCP connection so that congestion control can be dealt with in the 
>> >transport layer.
>>
>> connection if one already exists to the host,port and msrp 
>uri schema, 
>> rather reuse the existing connection.
>

>From application layer, use chunking for congestion control.

Sanjay

---

From: Rémi Denis-Courmont <[email-masked]>
Date: Wed, 12 Dec 2007 15:55:38 +0200

Le Wednesday 12 December 2007 15:10:20 ext Sanjay Sinha (sanjsinh), vous avez 
> From application layer, use chunking for congestion control.

Say alice.acme sends a large file to bob.corp; both use their own relay:

alice.acme ---> relay.acme ---> relay.corp ---> bob.corp

Further say that calin.acme sends another large file to donald.corp:

calin.acme ---> relay.acme ---> relay.corp ---> donald.corp

It would seem that relay.acme and relay.corp will use a single connection for 
both flows, so effectively, we have the following TCP connections:

alice.acme -----\                        /----- bob.corp
                relay.acme ---> relay.corp
calin.acme -----/                        \----- donald.corp

Now lets assume the following bandwidths (in kbytes/s for instance):

Alice to relay:   1000
Calin to relay:   1000
Relay to relay: 100000
Relay to bob:     2000
Relay to donald:    10

Alice and Calin will send data at speed 1000.
Relay.acme will chunk and forward to relay.corp at a cumulated rate of 2000.
Then relay.corp will start sending to Bob at 1000, and Donald also.

Obviously, the TCP buffer between Donald and his relay overflows quickly. 
However, Calin and relay.acme are blissfully unaware of the problem this far, 
so they keep sending data. Soon an extra chunk arrives at relay.corp for 
Donald...

Now what does relay.corp do? Basically, it can:
1/ put the connection to relay.acme on hold until the connection to 
donald.corp can accept more data, or
2/ discard the chunk, or
3/ queue chunk to some storage ad-nauseam, or
4/ refuse the chunk (return an error to relay.corp),

In case 1, chunks from Alice will also be blocked, since a TCP connection 
works as a serialized stream. Essentially, the session between Alice and Bob 
will be put down to a speed of 10, which is REALLY BAD.

In case 2, the session between Calin and Donald will eventually time out due 
to missing chunks (keep in mind MSRP has no negative acknowledgment 
mechanism).

In case 3, relay.corp needs virtually infinite storage - even then, Calin 
would consider the session has timed out way before all data has been 
dequeued out of relay.corp to Donald.

In case 4, the session between Calin and Donald will fail immediately.

Should Alice and Bob be inconvenienced by the sluggishness of Donald? or 
should Calin be prevented from sending large files to Donald just because 
Alice and Bob are faster?

Sorry, but I totally fail to see how chunking solves the problem. Probably 
because it does not.

Before someones discard this scenario has a weird corner case, I would like to 
point out that:
- If Donald is evil (possibly with cooperation from Bob), this becomes a 
trivial DoS attack.
- The statistical spread of bandwidth values increases with the number of 
simultaneous users of a single relay, such that the probability that such a 
problem occurs converges to 1.0 pretty fast.

Or then, we must declare that MSRP is only suited for "small" messages, 
and "file transfers" are not allowed. I'd rather use email then though.

-- 
Rémi Denis-Courmont

---

From: "Sanjay Sinha (sanjsinh)" <[email-masked]>
Date: Wed, 12 Dec 2007 11:20:27 -0500

Pl. see inline .. 

>From: Rémi Denis-Courmont [mailto:[email-masked]] 
>Sent: Wednesday, December 12, 2007 8:56 AM
>To: [email-masked]
>Le Wednesday 12 December 2007 15:10:20 ext Sanjay Sinha 
>It would seem that relay.acme and relay.corp will use a single 
>connection for both flows, so effectively, we have the 
>following TCP connections:
>
>However, Calin and relay.acme are blissfully unaware of the 
>problem this far, so they keep sending data. Soon an extra 
>chunk arrives at relay.corp for Donald...
>
>1/ put the connection to relay.acme on hold until the 
>connection to donald.corp can accept more data, or 2/ discard 
>the chunk, or 3/ queue chunk to some storage ad-nauseam, or 4/ 
>refuse the chunk (return an error to relay.corp),
>
>In case 1, chunks from Alice will also be blocked, since a TCP 
>connection works as a serialized stream. Essentially, the 
>session between Alice and Bob will be put down to a speed of 
>10, which is REALLY BAD.
>
>In case 2, the session between Calin and Donald will 
>eventually time out due to missing chunks (keep in mind MSRP 
>has no negative acknowledgment mechanism).

Not sure what you mean. Can't relay.corp send a 503 error response with possibly a Retry-After header for the session from calin once it relises that the connection to donald is slow and it should not affect punish alice --> bob communication because of donald's slow link.

>
>In case 3, relay.corp needs virtually infinite storage - even 
>then, Calin would consider the session has timed out way 
>before all data has been dequeued out of relay.corp to Donald.
>

Again I do not think sending a 503 response with a Retry-After header will cause Calin - Donald session to fail.

Another option for relay.acme, when it is multiplexing multiple sessions on a connection, is to open a new connection to relay.corp when it gets 503 response for the slow session and service Alice - Bob communication on that connection. Connection-reuse is recommended in RFCs, but it does not prohibit opening new connections for these kind of flow cases.

Thanks
Sanjay.

>
>Should Alice and Bob be inconvenienced by the sluggishness of 
>Donald? or should Calin be prevented from sending large files 
>to Donald just because Alice and Bob are faster?
>
>
>Sorry, but I totally fail to see how chunking solves the 
>problem. Probably because it does not.
>
>

---

From: Paul Kyzivat <[email-masked]>
Date: Wed, 12 Dec 2007 18:07:56 -0500

Its been a very long time, but some memories of discussions on this 
subject are finally coming back to me...

As I recall, there were in fact extensive discussions on this and 
related topics. IIRC we ultimately concluded that we could not solve 
every problem and so agreed to give up on solving this one. I think the 
assumption was that the relays could be assumed to have a lot of 
resources, such as the ability to buffer to disk to mitigate this problem.

I know that's not a very good answer, but its what I remember. I may 
however be remembering wrongly. Hopefully somebody else can confirm and 
fill in details, or else provide a different answer.

	Paul

---

From: Rémi Denis-Courmont <[email-masked]>
Date: Thu, 13 Dec 2007 10:10:28 +0200

Le Wednesday 12 December 2007 18:20:27 ext Sanjay Sinha (sanjsinh), vous avez 
écrit :
> Not sure what you mean. Can't relay.corp send a 503 error response with
> possibly a Retry-After header for the session from calin once it relises
> that the connection to donald is slow and it should not affect punish alice
> --> bob communication because of donald's slow link.

The 503 what?
http://www.iana.org/assignments/msrp-parameters

/!\ MSRP does not share its error codes space with HTTP.

> Another option for relay.acme, when it is multiplexing multiple sessions on
> a connection, is to open a new connection to relay.corp when it gets 503
> response for the slow session and service Alice - Bob communication on that
> connection. Connection-reuse is recommended in RFCs, but it does not
> prohibit opening new connections for these kind of flow cases.

There are only two 5xx error codes, none of which has such semantics:
501    Request-method-not-understood                  [RFC4975]
506    Session-already-in-use                         [RFC4975]

-- 
Rémi Denis-Courmont

---

From: Ruud Klaver <[email-masked]>
Date: Thu, 13 Dec 2007 09:21:12 +0100

>> Subject: Re: [Simple] MSRP congestion control issue when using relays

Rémi, thank you for the comprehensive example and explanation.

>> session between Alice and Bob will be put down to a speed of

This is indeed what I meant with my case (b).

>> has no negative acknowledgment mechanism).

The only difference between your case 2 and 4 is that for case 4 the  
original sender actually gets notified that the chunk was dropped  
with some error message. Is this correct?

MSRP however does have *optional* selective acknowledgments, which  
the sender could use to glean information about chunks received. But  
as I said, we do not want to go down that road, since it's TCP all  
over again.

> Not sure what you mean. Can't relay.corp send a 503 error response  
> with possibly a Retry-After header for the session from calin once  
> it relises that the connection to donald is slow and it should not  
> affect punish alice --> bob communication because of donald's slow  
> link.

Neither a 503 response or a Retry-After header are specified in the  
MSRP RFC. Even if they were specified it would have some major  
drawbacks:
- It assumes well-behaved clients, see the DoS issue Rémi mentioned.
- It actually drops data, which implies that the original sender has  
to do retransmissions and probably keep track of some form of sliding  
window. Are these terms (sliding window, negative acknowledgments,  
etc.) in any way familiar? They should be, as they are usually  
associated with TCP or other transport layer protocols.

>> before all data has been dequeued out of relay.corp to Donald.

I had not even considered this option as infinite storage is just  
plain impossible.

> Again I do not think sending a 503 response with a Retry-After  
> header will cause Calin - Donald session to fail.
>
> Another option for relay.acme, when it is multiplexing multiple  
> sessions on a connection, is to open a new connection to relay.corp  
> when it gets 503 response for the slow session and service Alice -  
> Bob communication on that connection. Connection-reuse is  
> recommended in RFCs, but it does not prohibit opening new  
> connections for these kind of flow cases.
>
> Thanks
> Sanjay.

This sounds unnecessarily complex to me and addresses neither of the  
issues I mentioned above.

Also, I'm having major deja vu, as pretty much everything you  
mentioned was already discussed back in 2003 and there is really no  
sense in regurgitating it here. Part of the intention of my original  
mail was to find out how this discussion was resolved eventually and  
if/how this is reflected in the RFCs.

In any case I would like to discuss the drawbacks of using a single  
TCP connection per session. The most prominent one I can think of  
right now is the overhead in negotiating a TLS session each time, as  
TLS is mandated when using relays. Anything else?

Ruud Klaver

---

From: Ruud Klaver <[email-masked]>
Date: Thu, 13 Dec 2007 09:33:49 +0100

On Dec 13, 2007, at 12:07 AM, Paul Kyzivat wrote:

> related topics. IIRC we ultimately concluded that we could not  
> solve every problem and so agreed to give up on solving this one. I  
> think the assumption was that the relays could be assumed to have a  
> lot of resources, such as the ability to buffer to disk to mitigate  
> this problem.
>
> I know that's not a very good answer, but its what I remember. I  
> may however be remembering wrongly. Hopefully somebody else can  
> confirm and fill in details, or else provide a different answer.
>
> 	Paul

Thanks for responding as someone who took part in the original  
discussion. As I said, I tried reading up on it in the archives, but  
I'm sure I didn't catch all of it.

As an implementor I am mostly interested in a practical solution and  
getting something that works. Assuming infinite storage does not  
sound very practical to me. Using one TCP connection per session  
sounds like a much more plausible compromise.

Of course I can just go ahead and implement it any way I want, but I  
really wanted to raise this issue in the interest of  
interoperability, since the RFCs are just unclear about this.

Ruud Klaver

---

From: "Sanjay Sinha (sanjsinh)" <[email-masked]>
Date: Thu, 13 Dec 2007 10:02:16 -0500

Inline ...

>Sent: Thursday, December 13, 2007 3:10 AM
>To: Sanjay Sinha (sanjsinh)
>Cc: [email-masked]
>Le Wednesday 12 December 2007 18:20:27 ext Sanjay Sinha 
>eventually time 
>> >out due to missing chunks (keep in mind MSRP has no negative 
>> >acknowledgment mechanism).
>>
>> with possibly a Retry-After header for the session from 
>calin once it 
>> relises that the connection to donald is slow and it should 
>not affect 
>> punish alice

I am not sure if there are any limitations around adding 503 error code to that list, but looks like it is something which may be useful for your case.

Thanks
Sanjay

>
>> Again I do not think sending a 503 response with a 
>Retry-After header 
>> will cause Calin - Donald session to fail.
>>
>> when it gets 503 response for the slow session and service 
>Alice - Bob 
>> communication on that connection. Connection-reuse is recommended in 
>> RFCs, but it does not prohibit opening new connections for 
>these kind of flow cases.
>

---

From: "Sanjay Sinha (sanjsinh)" <[email-masked]>
Date: Thu, 13 Dec 2007 10:16:50 -0500

Pl. see inline ...

>Sent: Thursday, December 13, 2007 3:21 AM
>To: [email-masked]
>>> donald.corp can accept more data, or 2/ discard the chunk, or 3/ 
>>> queue chunk to some storage ad-nauseam, or 4/ refuse the chunk 
>>> (return an error to relay.corp),
>>>
>>> connection works as a serialized stream. Essentially, the session 
>>> between Alice and Bob will be put down to a speed of 10, which is 
>>> REALLY BAD.
>
>The only difference between your case 2 and 4 is that for case 
>4 the original sender actually gets notified that the chunk 
>was dropped with some error message. Is this correct?
>
>MSRP however does have *optional* selective acknowledgments, 
>which the sender could use to glean information about chunks 
>received. But as I said, we do not want to go down that road, 
>since it's TCP all over again.
>
>> punish alice --> bob communication because of donald's slow link.
>
>Neither a 503 response or a Retry-After header are specified 
>in the MSRP RFC. Even if they were specified it would have some major
>drawbacks:

I was saying about using 503 from relay to sender, not from receiver to relay/sender. I know that only well-behaved clients may send 503.

>- It actually drops data, which implies that the original 
>sender has to do retransmissions and probably keep track of 
>some form of sliding window. Are these terms (sliding window, 
>negative acknowledgments,

Any kind of negative response will cause data drop or possibly affect the session. This kind of response atleast gives the sender an option to retry after the interval mentioned in Retry-After header.
>
>>>
>>> In case 3, relay.corp needs virtually infinite storage - even then, 
>>> Calin would consider the session has timed out way before all data 
>>> has been dequeued out of relay.corp to Donald.
>
>I had not even considered this option as infinite storage is 
>just plain impossible.
>
>these kind 
>> of flow cases.
>This sounds unnecessarily complex to me and addresses neither 
>of the issues I mentioned above.
>
>mentioned was already discussed back in 2003 and there is 
>really no sense in regurgitating it here. Part of the 
>intention of my original mail was to find out how this 
>discussion was resolved eventually and if/how this is 
>reflected in the RFCs.
>
>In any case I would like to discuss the drawbacks of using a 
>single TCP connection per session. The most prominent one I 
>can think of right now is the overhead in negotiating a TLS 
>session each time, as TLS is mandated when using relays. Anything else?

Do you mean not using a single connection per session? It is recommended in the RFC to reuse the connection, it does not mandate an implementation to do that. If you think that your implementation needs to use a separate connection per share, I think you can do that. The drawbacks are that you will be using more resources per session as well as overhead in negotiation with TLS, as you mentioned.

Sanjay

---

From: Robert Sparks <[email-masked]>
Date: Thu, 13 Dec 2007 14:46:56 -0600

My memory (after dredging it for a few days) recalls that we ended up  
with agreement that there were many ways to handle these kinds of  
problems and that we would develop some experience around them.
Having one connection between relays was something explicitly agreed  
to. Relays were expected to have significant, but not infinite  
storage, and how the relay chose to manage that resource was its own  
business.

IIRC the question on having a slow outgoing leg back up the incoming  
legs was put down when we realized the relay can just close that  
outgoing leg and tell the incoming side of whatever was going down  
that outgoing leg that the session's over. Other existing sessions  
coming through those incoming pipes and going down other outgoing  
pipes won't be affected. (Rohan - you may want to chime in here).

RjS

On Dec 13, 2007, at 2:33 AM, Ruud Klaver wrote:

>> Its been a very long time, but some memories of discussions on  
>> this subject are finally coming back to me...
>>
>> solve every problem and so agreed to give up on solving this one.  
>> I think the assumption was that the relays could be assumed to  
>> have a lot of resources, such as the ability to buffer to disk to  
>> mitigate this problem.
>>
> discussion. As I said, I tried reading up on it in the archives,  
> but I'm sure I didn't catch all of it.
>
> As an implementor I am mostly interested in a practical solution  
> and getting something that works. Assuming infinite storage does  
> not sound very practical to me. Using one TCP connection per  
> session sounds like a much more plausible compromise.
>
> Of course I can just go ahead and implement it any way I want, but  
> I really wanted to raise this issue in the interest of