Subject: Re: [tcpm] TCP evolution impact on router buffers

From: "Scharf, Michael (Michael)" <[email-masked]>
Date: Fri, 2 Oct 2015 17:05:49 +0000

Regarding Section 7 in [2], I am surprised that a potential impact of CIR/PIR is not mentioned. Unfortunately, I have only a very limited understanding of MPLS/IP router architectures, but the way bursts hit an AQM could IMHO depend on that.

As [2] is mentioned, one question related to the discussion below. From [2]:

3.2.  Buffer size

   The size of the buffers should be carefully chosen, and is to be set
   to the bandwidth-delay product; the bandwidth being the bottleneck
   capacity and the delay the largest RTT in the considered network.

This means of the order of 10GB of buffer size for a 400G router interface, right? Wouldn't it make sense to relax the statement "... and is to be set..." a bit?

Michael

From: Kuhn Nicolas [mailto:[email-masked]]
Sent: Wednesday, September 30, 2015 6:41 PM
To: Lane Wigley (lwigley); Scharf, Michael (Michael); [email-masked]
Subject: RE: TCP evolution impact on router buffers

Hi,

In [1], the authors evaluate the performance of CUBIC in small buffer regime. However, they use the Linux kernel version 2.6.18 (where the IW was 3 and not 10). Also, for any tests on CUBIC, small buffers and IW3/10, you may be interested in looking at the scenario "7.  Burst absorption" of the AQM Characterization Guidelines of the AQM WG [2]. Concerning TCP pacing, there is an on-going document on a pacing solution in the slow-start that you may want to have a look and comment [3].

I hope this helps,
Cheers,

Nicolas

[1] Jain, S.; Raina, G., "An experimental evaluation of CUBIC TCP in a small buffer regime," in Communications (NCC), 2011 National Conference on , vol., no., pp.1-5, 28-30 Jan. 2011
doi: 10.1109/NCC.2011.5734779
URL: http://ieeexplore.ieee.org/stamp/stamp.jsp?tp=&arnumber=5734779&isnumber=5734693
[2] https://tools.ietf.org/html/draft-ietf-aqm-eval-guidelines-08
[3] https://tools.ietf.org/html/draft-sallantin-tcpm-initial-spreading-01

De : tcpm [mailto:[email-masked]] De la part de Lane Wigley (lwigley)
Envoyé : mercredi 30 septembre 2015 15:10
À : Scharf, Michael (Michael); [email-masked]<mailto:[email-masked]>
Objet : Re: [tcpm] TCP evolution impact on router buffers

This paper from 2008 shows no drops with a 5 msec buffer on a heavily-congested core link. Low levels of drops start to show up around 2.5 msec. This was conducted at Level 3 to test the BDP / sqrt # flows proposal from Stanford.

Experimental Study of Router Buffer Sizing - Neda Beheshti, Yashar Ganjali, Monia Ghobadi, Nick McKeown, and Geoff Salmon

They also explored TCP pacing as a way to further reduce buffers. I'm trying to assess if CUBIC results in some pacing effect by decoupling the window growth from acks.

- Lane

From: Scharf, Michael (Michael) [mailto:[email-masked]]
Sent: Tuesday, September 29, 2015 6:56 PM
To: Lane Wigley (lwigley); [email-masked]<mailto:[email-masked]>
Subject: RE: TCP evolution impact on router buffers

This question may also be in scope of the AQM WG, and perhaps ICCRG, but since the communities overlap, I avoid cross-posting...

Unfortunately, I don't have any recent pointer or own data. But I am aware of examples for router buffer dimensioning based on tests with a single TCP connection, using Reno. In those examples, the dimension of the buffers and possibly other related parameters (two-color/three-color meters/policiers, WRED slopes, H-QoS, etc.) are configured to achieve full "line" speed for a single TCP connection. As to be expected, this typically comes down to the old BDP rule of thumb. But it is rather obvious that a single bulk data Reno connection is not necessarily a realistic workload these days. So, the test methodology can be one relevant aspect of the dimensioning problem.

I believe that a modern TCP stack with a high-speed congestion control such as CUBIC will typically need less buffer than the outcome of this dimensioning method and the publications 10 years ago. As far as I know, most modern TCP stacks operate well over a very wide range of buffer sizes, and the buffers can be relatively small. To me, the key question is the latency-throughput tradeoff, and there may not be a one-fits-all answer to that one.

Personally, I'd also be interested in more recent pointers on router buffer dimensioning. I could even see some value in documenting dimensioning guidelines used in practice, if there was a way to generalize them.

Michael

From: tcpm [mailto:[email-masked]] On Behalf Of Lane Wigley (lwigley)
Sent: Friday, September 25, 2015 5:41 PM
To: [email-masked]<mailto:[email-masked]>
Subject: [tcpm] TCP evolution impact on router buffers

I'm trying to track down some research or opinions on how the more recent TCP schemes impact router buffer needs. There are a number of publications from Stanford and Georgia Tech from about 10 years ago, and I'm trying to assess how changes to the algorithms (e.g. CUBIC) and parameters (initial congestion window) deployed since then may influence those findings in the direction of more or less buffering being needed.

I'd appreciate any input and pointers. Thanks.

- Lane

---

From: Veaceslav ROMAN <[email-masked]>
Date: Mon, 5 Oct 2015 22:54:26 +0000

Can be 400G router interface considered as a bottleneck capacity if all downstream receivers are at a maximum of 1G ?

Best regards

Veaceslav Roman

________________________________
From: tcpm [[email-masked]] on behalf of Scharf, Michael (Michael) [[email-masked]]
Sent: Friday, October 02, 2015 8:05 PM
To: Kuhn Nicolas; Lane Wigley (lwigley); [email-masked]
Subject: Re: [tcpm] TCP evolution impact on router buffers

3.2.  Buffer size

This means of the order of 10GB of buffer size for a 400G router interface, right? Wouldn’t it make sense to relax the statement “… and is to be set…” a bit?

Michael

Hi,

In [1], the authors evaluate the performance of CUBIC in small buffer regime. However, they use the Linux kernel version 2.6.18 (where the IW was 3 and not 10). Also, for any tests on CUBIC, small buffers and IW3/10, you may be interested in looking at the scenario “7.  Burst absorption” of the AQM Characterization Guidelines of the AQM WG [2]. Concerning TCP pacing, there is an on-going document on a pacing solution in the slow-start that you may want to have a look and comment [3].

I hope this helps,
Cheers,

Nicolas

Experimental Study of Router Buffer Sizing – Neda Beheshti, Yashar Ganjali, Monia Ghobadi, Nick McKeown, and Geoff Salmon

They also explored TCP pacing as a way to further reduce buffers. I’m trying to assess if CUBIC results in some pacing effect by decoupling the window growth from acks.

- Lane

This question may also be in scope of the AQM WG, and perhaps ICCRG, but since the communities overlap, I avoid cross-posting…

Unfortunately, I don’t have any recent pointer or own data. But I am aware of examples for router buffer dimensioning based on tests with a single TCP connection, using Reno. In those examples, the dimension of the buffers and possibly other related parameters (two-color/three-color meters/policiers, WRED slopes, H-QoS, etc.) are configured to achieve full “line” speed for a single TCP connection. As to be expected, this typically comes down to the old BDP rule of thumb. But it is rather obvious that a single bulk data Reno connection is not necessarily a realistic workload these days. So, the test methodology can be one relevant aspect of the dimensioning problem.

Personally, I’d also be interested in more recent pointers on router buffer dimensioning. I could even see some value in documenting dimensioning guidelines used in practice, if there was a way to generalize them.

Michael

I’m trying to track down some research or opinions on how the more recent TCP schemes impact router buffer needs. There are a number of publications from Stanford and Georgia Tech from about 10 years ago, and I’m trying to assess how changes to the algorithms (e.g. CUBIC) and parameters (initial congestion window) deployed since then may influence those findings in the direction of more or less buffering being needed.

I’d appreciate any input and pointers. Thanks.

- Lane

---

From: "Scharf, Michael (Michael)" <[email-masked]>
Date: Mon, 5 Oct 2015 23:10:55 +0000

What prevents congestion in the case you mention? E.g., what happens if there significantly more than 400 customers/customer ports with 1G each?

I don't know whether such a fast interface really benefits from AQM, and whether interfaces of that order of magnitude are in scope of document [2], or not. But just having a very fast link does not imply that congestion can never occur. Otherwise TCPM's job would be much simpler ;)

Michael

From: Veaceslav ROMAN [mailto:[email-masked]]
Sent: Tuesday, October 06, 2015 12:54 AM
To: Scharf, Michael (Michael); Kuhn Nicolas; Lane Wigley (lwigley); [email-masked]
Subject: RE: TCP evolution impact on router buffers

Best regards

Veaceslav Roman

To: Kuhn Nicolas; Lane Wigley (lwigley); [email-masked]<mailto:[email-masked]>

3.2.  Buffer size

Michael

From: Kuhn Nicolas [mailto:[email-masked]]
Sent: Wednesday, September 30, 2015 6:41 PM
To: Lane Wigley (lwigley); Scharf, Michael (Michael); [email-masked]<mailto:[email-masked]>
Subject: RE: TCP evolution impact on router buffers

Hi,

I hope this helps,
Cheers,

Nicolas

- Lane

Michael

- Lane

---

From: Veaceslav ROMAN <[email-masked]>
Date: Tue, 6 Oct 2015 06:17:58 +0000

Sure, congestion may happen even at 400. The question is what B one should use in BDP estimation for buffer sizing and if BDP approach is the right one in conditions when individual stream is physically limited to much lower spead?

Sent from my Samsung device

-------- Original message --------
From: "Scharf, Michael (Michael)" <[email-masked]>
Date: 2015/10/06 02:10 (GMT+02:00)
To: Veaceslav ROMAN <[email-masked]>, Kuhn Nicolas <[email-masked]>, "Lane Wigley (lwigley)" <[email-masked]>, [email-masked]
Subject: RE: TCP evolution impact on router buffers

I don’t know whether such a fast interface really benefits from AQM, and whether interfaces of that order of magnitude are in scope of document [2], or not. But just having a very fast link does not imply that congestion can never occur. Otherwise TCPM’s job would be much simpler ;)

Michael

Best regards

Veaceslav Roman

3.2.  Buffer size

Michael

Hi,

I hope this helps,
Cheers,

Nicolas

- Lane

Michael

- Lane

---

From: Kuhn Nicolas <[email-masked]>
Date: Wed, 7 Oct 2015 10:16:30 +0000

Hi,

1-      Regarding the CIR/PIR

IMHO this is out of the scope of the characterization guidelines. However, if you think that we should introduce more information on the MPLS/IP router architectures please let us know. The only problem being that we want to be quite generic and do not want to focus on specific architectures.

2-      Regarding buffer sizes

Since buffer sizing is not the only problem of evaluating AQMs, we have looked for RFC clearly stipulating how the "adequate" buffer size can be found. As said in RFC7597:
"
The optimum buffer size is a function of
   operational requirements and should generally be sized to be
   sufficient to buffer the largest normal traffic burst that is
   expected.  This size depends on the amount and burstiness of traffic
   arriving at the queue and the rate at which traffic leaves the queue.
"

In the characterization draft, the recommendation on the buffer size is not a "MUST".
If you think that we should more clearly say that the size of the buffer MAY be set to the BDP, let us know.
In its current state, it has been voluntary that the draft does not impose any buffer size.

Cheers,

Nicolas

De : Scharf, Michael (Michael) [mailto:[email-masked]]
Envoyé : vendredi 2 octobre 2015 19:06
À : Kuhn Nicolas; Lane Wigley (lwigley); [email-masked]
Objet : RE: TCP evolution impact on router buffers

3.2.  Buffer size

Michael

Hi,

I hope this helps,
Cheers,

Nicolas

- Lane

Michael

- Lane

---

From: "Scharf, Michael (Michael)" <[email-masked]>
Date: Wed, 7 Oct 2015 11:35:25 +0000

Regarding 1: Sure, it doesn't make any sense to add architecture-specific guidelines. I just wonder if the text could mention a bit more explicitly that a number of potential router components and algorithms (e.g., DiffServ) are omitted (or abstracted) to keep the guidelines generic and simple.

Regarding 2: Yes, given the references mentioned by Lane, I'd suggest to somehow replace the wording "... and is to be set to the bandwidth-delay product", e.g., by a MAY.

But I leave this to the authors - this is not really in TCPM scope, and I don't have cycles to follow the AQM WG.

Michael

From: Kuhn Nicolas [mailto:[email-masked]]
Sent: Wednesday, October 07, 2015 12:17 PM
To: Scharf, Michael (Michael); Lane Wigley (lwigley); [email-masked]
Subject: RE: TCP evolution impact on router buffers

Hi,

1-      Regarding the CIR/PIR

Cheers,

Nicolas

De : Scharf, Michael (Michael) [mailto:[email-masked]]
Envoyé : vendredi 2 octobre 2015 19:06
À : Kuhn Nicolas; Lane Wigley (lwigley); [email-masked]<mailto:[email-masked]>
Objet : RE: TCP evolution impact on router buffers

3.2.  Buffer size

Michael

Hi,

I hope this helps,
Cheers,

Nicolas

- Lane

Michael

- Lane

---

From: "Lane Wigley (lwigley)" <[email-masked]>
Date: Thu, 8 Oct 2015 14:41:03 +0000

Just to wrap this thread up, I found a paper that addressed my original question. It looked at the impact of mixing different TCP implementations and one of the calculations was the standard deviation of the congestion window with various algorithms. All of the newer ones have lower standard deviation than Reno.

http://ieeexplore.ieee.org/stamp/stamp.jsp?arnumber=6742797

- Lane

From: Scharf, Michael (Michael) [mailto:[email-masked]]
Sent: Wednesday, October 07, 2015 7:35 AM
To: Kuhn Nicolas; Lane Wigley (lwigley); [email-masked]
Subject: RE: TCP evolution impact on router buffers

Michael

From: Kuhn Nicolas [mailto:[email-masked]]
Sent: Wednesday, October 07, 2015 12:17 PM
To: Scharf, Michael (Michael); Lane Wigley (lwigley); [email-masked]<mailto:[email-masked]>
Subject: RE: TCP evolution impact on router buffers

Hi,

1-     Regarding the CIR/PIR

Cheers,

Nicolas

3.2.  Buffer size

Michael

Hi,

I hope this helps,
Cheers,

Nicolas

- Lane

Michael

- Lane

---

From: Mirja Kühlewind <[email-masked]>
Date: Fri, 9 Oct 2015 11:19:59 +0200

Hi all,

one more comment. How much buffer is need mostly depends on the decrease 
factor of your congestion control which is 0.5 for Reno (and many others) and 
0.7 for Cubic. That means Cubic reduces less on congestion and therefore 
needs less buffer to keep the link full during the time period where the 
window was reduced. This is easy to calculate if there is only one flow. Of 
course if you have different flows that are not synchronized and might use 
different schemes, it's more complicated. And then also the increase behavior 
influences how much you'll underutilize the link if the buffer is 'too small'.

However, the actual point I wanted to make it that most schemes use fixed 
decrease values which (at least theoretically) induce a dependency on the 
buffer size. There is e.g. H-TCP (and also a scheme I've proposed and 
presented a while ago at an ICCRG meeting called TCP SIAD) that calculate the 
decrease factor dynamically based on RTT measurements. These schemes can 
basically work well with any buffer size, however, of course they are 
slightly more complicated and as there was not really a big need for them up 
to now, they are not really deployed.

Mirja

On 08.10.2015 16:41, Lane Wigley (lwigley) wrote:
> Just to wrap this thread up, I found a paper that addressed my original
> question. It looked at the impact of mixing different TCP implementations and
> one of the calculations was the standard deviation of the congestion window
> with various algorithms. All of the newer ones have lower standard deviation
> than Reno.
>
> *From:*Scharf, Michael (Michael) [mailto:[email-masked]]
> *Sent:* Wednesday, October 07, 2015 7:35 AM
> *To:* Kuhn Nicolas; Lane Wigley (lwigley); [email-masked]
> *Subject:* RE: TCP evolution impact on router buffers
>
> Regarding 1: Sure, it doesn’t make any sense to add architecture-specific
> guidelines. I just wonder if the text could mention a bit more explicitly
> that a number of potential router components and algorithms (e.g., DiffServ)
> are omitted (or abstracted) to keep the guidelines generic and simple.
>
> Regarding 2: Yes, given the references mentioned by Lane, I’d suggest to
> somehow replace the wording “… and is to be set to the bandwidth-delay
> product”, e.g., by a MAY.
>
> But I leave this to the authors – this is not really in TCPM scope, and I
> don’t have cycles to follow the AQM WG.
>
> Michael
>
> *From:*Kuhn Nicolas [mailto:[email-masked]]
> *Sent:* Wednesday, October 07, 2015 12:17 PM
> *To:* Scharf, Michael (Michael); Lane Wigley (lwigley); [email-masked]
> <mailto:[email-masked]>
> *Subject:* RE: TCP evolution impact on router buffers
>
> Hi,
>
> 1-Regarding the CIR/PIR
>
> IMHO this is out of the scope of the characterization guidelines. However, if
> you think that we should introduce more information on the MPLS/IP router
> architectures please let us know. The only problem being that we want to be
> quite generic and do not want to focus on specific architectures.
>
> 2-Regarding buffer sizes
>
> Since buffer sizing is not the only problem of evaluating AQMs, we have
> looked for RFC clearly stipulating how the “adequate” buffer size can be
> found. As said in RFC7597:
>
> “
>
> “
> In the characterization draft, the recommendation on the buffer size is not a
> “MUST”.
>
> *De :*Scharf, Michael (Michael) [mailto:[email-masked]]
> *Envoyé :* vendredi 2 octobre 2015 19:06
> *À :* Kuhn Nicolas; Lane Wigley (lwigley); [email-masked] <mailto:[email-masked]>
> *Objet :* RE: TCP evolution impact on router buffers
>
> Regarding Section 7 in [2], I am surprised that a potential impact of CIR/PIR
> is not mentioned. Unfortunately, I have only a very limited understanding of
> MPLS/IP router architectures, but the way bursts hit an AQM could IMHO depend
> on that.
>
> This means of the order of 10GB of buffer size for a 400G router interface,
> right? Wouldn’t it make sense to relax the statement “… and is to be set…” a bit?
>
> Michael
>
> *From:*Kuhn Nicolas [mailto:[email-masked]]
> *Sent:* Wednesday, September 30, 2015 6:41 PM
> *To:* Lane Wigley (lwigley); Scharf, Michael (Michael); [email-masked]
> <mailto:[email-masked]>
> *Subject:* RE: TCP evolution impact on router buffers
>
> Hi,
>
> In [1], the authors evaluate the performance of CUBIC in small buffer regime.
> However, they use the Linux kernel version 2.6.18 (where the IW was 3 and not
> 10). Also, for any tests on CUBIC, small buffers and IW3/10, you may be
> interested in looking at the scenario “7.  Burst absorption” of the AQM
> Characterization Guidelines of the AQM WG [2]. Concerning TCP pacing, there
> is an on-going document on a pacing solution in the slow-start that you may
> want to have a look and comment [3].
>
> I hope this helps,
>
> Cheers,
>
> Nicolas
>
> [1] Jain, S.; Raina, G., "An experimental evaluation of CUBIC TCP in a small
> buffer regime," in /Communications (NCC), 2011 National Conference on/ ,
> vol., no., pp.1-5, 28-30 Jan. 2011
> URL:
> http://ieeexplore.ieee.org/stamp/stamp.jsp?tp=&arnumber=5734779&isnumber=5734693
>
> *De :*tcpm [mailto:[email-masked]] *De la part de* Lane Wigley (lwigley)
> *Envoyé :* mercredi 30 septembre 2015 15:10
> *À :* Scharf, Michael (Michael); [email-masked] <mailto:[email-masked]>
> *Objet :* Re: [tcpm] TCP evolution impact on router buffers
>
> This paper from 2008 shows no drops with a 5 msec buffer on a
> heavily-congested core link. Low levels of drops start to show up around 2.5
> msec. This was conducted at Level 3 to test the BDP / sqrt # flows proposal
> from Stanford.
>
> Experimental Study of Router Buffer Sizing – Neda Beheshti, Yashar Ganjali,
> Monia Ghobadi, Nick McKeown, and Geoff Salmon
>
> They also explored TCP pacing as a way to further reduce buffers. I’m trying
> to assess if CUBIC results in some pacing effect by decoupling the window
> growth from acks.
>
> - Lane
>
> *From:*Scharf, Michael (Michael) [mailto:[email-masked]]
> *Sent:* Tuesday, September 29, 2015 6:56 PM
> *To:* Lane Wigley (lwigley); [email-masked] <mailto:[email-masked]>
> *Subject:* RE: TCP evolution impact on router buffers
>
> This question may also be in scope of the AQM WG, and perhaps ICCRG, but
> since the communities overlap, I avoid cross-posting…
>
> Unfortunately, I don’t have any recent pointer or own data. But I am aware of
> examples for router buffer dimensioning based on tests with a single TCP
> connection, using Reno. In those examples, the dimension of the buffers and
> possibly other related parameters (two-color/three-color meters/policiers,
> WRED slopes, H-QoS, etc.) are configured to achieve full “line” speed for a
> single TCP connection. As to be expected, this typically comes down to the
> old BDP rule of thumb. But it is rather obvious that a single bulk data Reno
> connection is not necessarily a realistic workload these days. So, the test
> methodology can be one relevant aspect of the dimensioning problem.
>
> I believe that a modern TCP stack with a high-speed congestion control such
> as CUBIC will typically need less buffer than the outcome of this
> dimensioning method and the publications 10 years ago. As far as I know, most
> modern TCP stacks operate well over a very wide range of buffer sizes, and
> the buffers can be relatively small. To me, the key question is the
> latency-throughput tradeoff, and there may not be a one-fits-all answer to
> that one.
>
> Personally, I’d also be interested in more recent pointers on router buffer
> dimensioning. I could even see some value in documenting dimensioning
> guidelines used in practice, if there was a way to generalize them.
>
> Michael
>
> *From:*tcpm [mailto:[email-masked]] *On Behalf Of *Lane Wigley (lwigley)
> *Sent:* Friday, September 25, 2015 5:41 PM
> *To:* [email-masked] <mailto:[email-masked]>
> *Subject:* [tcpm] TCP evolution impact on router buffers
>
> I’m trying to track down some research or opinions on how the more recent TCP
> schemes impact router buffer needs. There are a number of publications from
> Stanford and Georgia Tech from about 10 years ago, and I’m trying to assess
> how changes to the algorithms (e.g. CUBIC) and parameters (initial congestion
> window) deployed since then may influence those findings in the direction of
> more or less buffering being needed.
>
> _______________________________________________
> tcpm mailing list
> [email-masked]
> https://www.ietf.org/mailman/listinfo/tcpm
>