Subject: Re: [tcpm] TCP evolution impact on router buffers

From: "Lane Wigley (lwigley)" <[email-masked]>
Date: Tue, 6 Oct 2015 13:57:25 +0000

I think you want to use "B" of 400, but if this is an Internet link carrying TCP, this is a case where the Stanford work suggesting BDP / sqrt (# flows) [1] should apply as the TCP flows don't get highly synchronized when they are many of them. I'd want a bit of margin for error [2], but the trend since that work seems to be toward more smoothing of individual TCP sessions and thus less margin needed.

The key item from the first paper is that in 2008, they took 3 highly-congested links at Level 3 and changed the 190 msec BDP buffers to 10, 5, 2.5, and 1 msec. They didn't see any drops at 5 msec and saw 0.02% at 2.5 msec. With 10000 flows, this was inline with their predictions. After some back and forth with other researchers (including Georgia Tech), they suggested BDP / 10 as a conservative sizing.

[1] Behishti [www-afs.secure-endpoints.com/afs/cs.pitt.edu/usr/.../Buff-Sizing-Exp.pdf]
[2] Ganuali [sysweb.cs.toronto.edu/publication_files/.../Buffers-Update---CCR-06.pdf]

- Lane

> Date: Tue, 6 Oct 2015 06:17:58 +0000
> From: Veaceslav ROMAN <[email-masked]>
> To: "Scharf, Michael (Michael)" <[email-masked]>,
> 	"Kuhn Nicolas" <[email-masked]>, "Lane Wigley (lwigley)"
> 	<[email-masked]>,  "[email-masked]" <[email-masked]>
> Subject: Re: [tcpm] TCP evolution impact on router buffers
> Message-ID:
> <[email-masked]>
> Content-Type: text/plain; charset="windows-1252"
> 
> Sure, congestion may happen even at 400. The question is what B one
> should use in BDP estimation for buffer sizing and if BDP approach is the
> right one in conditions when individual stream is physically limited to much
> lower spead?
> 
> 
> 
> Sent from my Samsung device
> 
> 
> -------- Original message --------
> From: "Scharf, Michael (Michael)" <[email-masked]>
> Date: 2015/10/06 02:10 (GMT+02:00)
> To: Veaceslav ROMAN <[email-masked]>, Kuhn Nicolas
> <[email-masked]>, "Lane Wigley (lwigley)" <[email-masked]>,
> [email-masked]
> Subject: RE: TCP evolution impact on router buffers
> 
> What prevents congestion in the case you mention? E.g., what happens if
> there significantly more than 400 customers/customer ports with 1G each?
> 
> I don?t know whether such a fast interface really benefits from AQM, and
> whether interfaces of that order of magnitude are in scope of document [2],
> or not. But just having a very fast link does not imply that congestion can
> never occur. Otherwise TCPM?s job would be much simpler ;)
> 
> Michael
> 
> 
> From: Veaceslav ROMAN [mailto:[email-masked]]
> Sent: Tuesday, October 06, 2015 12:54 AM
> To: Scharf, Michael (Michael); Kuhn Nicolas; Lane Wigley (lwigley);
> [email-masked]
> Subject: RE: TCP evolution impact on router buffers
> 
> 
> Can be 400G router interface considered as a bottleneck capacity if all
> downstream receivers are at a maximum of 1G ?
> 
> 
> Best regards
> 
> Veaceslav Roman
> 
> 
> ________________________________
> From: tcpm [[email-masked]] on behalf of Scharf, Michael (Michael)
> [[email-masked]]
> Sent: Friday, October 02, 2015 8:05 PM
> To: Kuhn Nicolas; Lane Wigley (lwigley);
> [email-masked]<mailto:[email-masked]>
> Subject: Re: [tcpm] TCP evolution impact on router buffers Regarding Section
> 7 in [2], I am surprised that a potential impact of CIR/PIR is not mentioned.
> Unfortunately, I have only a very limited understanding of MPLS/IP router
> architectures, but the way bursts hit an AQM could IMHO depend on that.
> 
> As [2] is mentioned, one question related to the discussion below. From [2]:
> 
> 3.2.  Buffer size
> 
>    The size of the buffers should be carefully chosen, and is to be set
>    to the bandwidth-delay product; the bandwidth being the bottleneck
>    capacity and the delay the largest RTT in the considered network.
> 
> This means of the order of 10GB of buffer size for a 400G router interface,
> right? Wouldn?t it make sense to relax the statement ?? and is to be set?? a
> bit?
> 
> Michael
> 
> 
> From: Kuhn Nicolas [mailto:[email-masked]]
> Sent: Wednesday, September 30, 2015 6:41 PM
> To: Lane Wigley (lwigley); Scharf, Michael (Michael);
> [email-masked]<mailto:[email-masked]>
> Subject: RE: TCP evolution impact on router buffers
> 
> Hi,
> 
> In [1], the authors evaluate the performance of CUBIC in small buffer regime.
> However, they use the Linux kernel version 2.6.18 (where the IW was 3 and
> not 10). Also, for any tests on CUBIC, small buffers and IW3/10, you may be
> interested in looking at the scenario ?7.  Burst absorption? of the AQM
> Characterization Guidelines of the AQM WG [2]. Concerning TCP pacing,
> there is an on-going document on a pacing solution in the slow-start that
> you may want to have a look and comment [3].
> 
> I hope this helps,
> Cheers,
> 
> Nicolas
> 
> [1] Jain, S.; Raina, G., "An experimental evaluation of CUBIC TCP in a small
> buffer regime," in Communications (NCC), 2011 National Conference on ,
> vol., no., pp.1-5, 28-30 Jan. 2011
> doi: 10.1109/NCC.2011.5734779
> URL:
> http://ieeexplore.ieee.org/stamp/stamp.jsp?tp=&arnumber=5734779&isnum
> ber=5734693
> [2] https://tools.ietf.org/html/draft-ietf-aqm-eval-guidelines-08
> [3] https://tools.ietf.org/html/draft-sallantin-tcpm-initial-spreading-01
> 
> De : tcpm [mailto:[email-masked]] De la part de Lane Wigley (lwigley)
> Envoy? : mercredi 30 septembre 2015 15:10 ? : Scharf, Michael (Michael);
> [email-masked]<mailto:[email-masked]>
> Objet : Re: [tcpm] TCP evolution impact on router buffers
> 
> This paper from 2008 shows no drops with a 5 msec buffer on a heavily-
> congested core link. Low levels of drops start to show up around 2.5 msec.
> This was conducted at Level 3 to test the BDP / sqrt # flows proposal from
> Stanford.
> 
> Experimental Study of Router Buffer Sizing ? Neda Beheshti, Yashar Ganjali,
> Monia Ghobadi, Nick McKeown, and Geoff Salmon
> 
> They also explored TCP pacing as a way to further reduce buffers. I?m trying
> to assess if CUBIC results in some pacing effect by decoupling the window
> growth from acks.
> 
> - Lane
> 
> 
> From: Scharf, Michael (Michael) [mailto:[email-masked]]
> Sent: Tuesday, September 29, 2015 6:56 PM
> To: Lane Wigley (lwigley); [email-masked]<mailto:[email-masked]>
> Subject: RE: TCP evolution impact on router buffers
> 
> This question may also be in scope of the AQM WG, and perhaps ICCRG, but
> since the communities overlap, I avoid cross-posting?
> 
> Unfortunately, I don?t have any recent pointer or own data. But I am aware
> of examples for router buffer dimensioning based on tests with a single TCP
> connection, using Reno. In those examples, the dimension of the buffers and
> possibly other related parameters (two-color/three-color meters/policiers,
> WRED slopes, H-QoS, etc.) are configured to achieve full ?line? speed for a
> single TCP connection. As to be expected, this typically comes down to the
> old BDP rule of thumb. But it is rather obvious that a single bulk data Reno
> connection is not necessarily a realistic workload these days. So, the test
> methodology can be one relevant aspect of the dimensioning problem.
> 
> I believe that a modern TCP stack with a high-speed congestion control such
> as CUBIC will typically need less buffer than the outcome of this
> dimensioning method and the publications 10 years ago. As far as I know,
> most modern TCP stacks operate well over a very wide range of buffer sizes,
> and the buffers can be relatively small. To me, the key question is the
> latency-throughput tradeoff, and there may not be a one-fits-all answer to
> that one.
> 
> Personally, I?d also be interested in more recent pointers on router buffer
> dimensioning. I could even see some value in documenting dimensioning
> guidelines used in practice, if there was a way to generalize them.
> 
> Michael
> 
> 
> 
> From: tcpm [mailto:[email-masked]] On Behalf Of Lane Wigley
> (lwigley)
> Sent: Friday, September 25, 2015 5:41 PM
> To: [email-masked]<mailto:[email-masked]>
> Subject: [tcpm] TCP evolution impact on router buffers
> 
> I?m trying to track down some research or opinions on how the more recent
> TCP schemes impact router buffer needs. There are a number of publications
> from Stanford and Georgia Tech from about 10 years ago, and I?m trying to
> assess how changes to the algorithms (e.g. CUBIC) and parameters (initial
> congestion window) deployed since then may influence those findings in the
> direction of more or less buffering being needed.
> 
> I?d appreciate any input and pointers. Thanks.
> 
> - Lane
> 
> 
> -------------- next part --------------
> An HTML attachment was scrubbed...
> URL:
> <https://mailarchive.ietf.org/arch/browse/tcpm/attachments/20151006/156f
> a599/attachment.html>
> 
> ------------------------------
> 
> Subject: Digest Footer
> 
> _______________________________________________
> tcpm mailing list
> [email-masked]
> https://www.ietf.org/mailman/listinfo/tcpm
> 
> 
> ------------------------------
> 
> End of tcpm Digest, Vol 138, Issue 12
> *************************************