Subject: Re: [tcpm] [e2e] TCP HyStart patch deployment

From: Veaceslav ROMAN <[email-masked]>
Date: Thu, 1 Oct 2015 08:31:59 +0000

Yes, HYSTART_DELAY_MIN  (4U<<3) . 
But curr_rtt and delay_min come from bictcp_acked which is also <<3 : 
	In tcp_cubic.c, function  line 433 (kernel 4.2) 
		delay = (rtt_us << 3) / USEC_PER_MSEC;
So, 4U<<3 is equivalent to 4 ms .

Regarding pacing I am not sure will help the performance: there are gaps in ACK trains of the order of milliseconds and this is a lost time and there is no way to compensate it. If introduce ACK pacing or data packets pacing this would add to the overall lost time.
Pacing can help only to alleviate a potential problem of some downstream routers with insufficient buffers which may introduce unexpected packets drops when large train sent at a high speed as a reply to highly compressed ACK train, but, on another hands.
But how much pacing and how to accommodate to largely changing radio conditions ? And with permanently changing technology and speed ? Anyway the pacing will add to the problem of gaps and better size properly routers buffers. 

Ideally ACK train is split in 2 parts, but in practice I do see more. But as the stream progress the less gaps. The bigger the downlink pressure the more ACK and less chances to have a pause in ACK sending, then there will be a continuous flow of Buffer Status Reports and less of the Scheduling Request (SR, "Hey, I have some data to send"). The only issue I am not sure is on the potential gaps in other networks. I did not put bellow all details, but you should know that device can not send SR in arbitrary moments of time but in predefined for him intervals. In my network this period is 5 ms, therefore the RTT of the first ACK in the train will have variations of 2.5 ms. But standards admit this period to be also 10 ms ,20 ms, 40 ms, 80 ms and I can not say what is the practice of others. I may only guess that most will try to use the best and fastest, i.e. 5 ms.
Anyway, to get a realistic modeling of LTE interaction with TCP the detailed MAC scheduler model is needed. Unfortunately, most of the literature focus on scheduling of the Physical Resource Blocks, Adaptive Modulation and Coding, MIMO, i.e., focusing on the distribution of resources between users, which is good and needed, but do not address very much the timing interaction with TCP.  At least, I never met in LTE scheduling literature a term "TCP self-clocking". 

The 8 ms  HYSTART_DELAY_MIN could only be a simple solution for the problem of counting of the round' curr_rtt from the second ACK of the train (as the second ACK RTT will be, typically, due to scheduling, 5-7 ms longer than the first one), but this only if SR periodicity is 5 ms.  
The other 2 problems will remain unsolved.   

Veaceslav Roman

-----Original Message-----
From: Ingemar Johansson S [mailto:[email-masked]] 
Sent: Thursday, 01 October 2015 08:59
To: Veaceslav ROMAN; Eric Dumazet; Neal Cardwell
Cc: [email-masked]; Piers O'Hanlon; Sangtae Ha; Eric Dumazet; [email-masked]; Ingemar Johansson S
Subject: RE: [tcpm] [e2e] TCP HyStart patch deployment

Hi

Thanks for the detailed report. What seems unclear to me is the value of HYSTART_DELAY_MIN. You mention 8ms but when I look into the 4.2 kernel code I get the feeling that it is 32ms ( #define HYSTART_DELAY_MIN       (4U<<3)   ). Is this a mistake from my side  ?
I have run LTE simulations and tried to wring this HyStart issue inside out and upside down but sofar it has been very difficult to make HyStart exit prematurely but then of course I have run with HYSTART_DELAY_MIN = 32ms. 

As regards to the ACK-compression problem, yes, ACK compression easily occur in LTE, even in simulations. What I have seen is that packet pacing is a very efficient remedy, for instance the packet implemented in QUIC seems to solve ACK compression issues very well. 

It is interesting what you say that ACK trains are so much split up, I could understand that they are split up in two parts (initial grant + additional after received BSR).  

/Ingemar

> -----Original Message-----
> From: Veaceslav ROMAN [mailto:[email-masked]]
> Sent: den 1 oktober 2015 02:25
> To: Eric Dumazet; Neal Cardwell
> Cc: [email-masked]; Piers O'Hanlon; Sangtae Ha; Ingemar Johansson S; 
> Eric Dumazet; [email-masked]
> Subject: RE: [tcpm] [e2e] TCP HyStart patch deployment
> 
> Finally can share some results of Hystart interaction with LTE.
>  Few words about the testing configuration:
>  	Device Samsung Galaxy Note4LTE , up to 150 Mbps capable
>              Radio Network LTE 2600/1800
>              Distance between the Core Network (EPC) and Base Station - 10 km
>              HTTP Server connected directly to the Core Network 1 Gbps
>              Core network switching(EPC) 10 Gbps
>              Backhaul transport  1 Gbps, full IP, MPLS
>              Last mile transport full IP, MPLS,  300 Mbps
> 
>             Server: Linux k40srv 4.1.0-040100-generic #201507030940 
> SMP Fri Jul 3
> 09:41:47 UTC 2015 x86_64 x86_64 x86_64 GNU/Linux
>                         Apache/2.4.10 (Ubuntu), built:   Jul 24 2015 17:25:18
> 
>            Hystart_detect: 2 (HYSTART_DELAY only), as suggested.
>            net.ipv4.tcp_no_metrics_save = 1 to avoid tcp metrics 
> caching interference
> 
> 	Tests types:
> 		1. Download 3 MB file, interchanging Hystart ON and OFF, in 
> conditions of Low Bandwidth (25-35 Mbps, poor radio) and High 
> Bandwidth (100-120 Mbps, good radio).
> 		2. Download 10 MB file, interchanging Hystart ON and OFF, in 
> conditions of Low Bandwidth (25-35 Mbps, poor radio) and High 
> Bandwidth (100-120 Mbps, good radio).
> 
>  	100 Download tests per type.
>              Download duration and bandwidth measured at the client 
> side as shown by curl from the first received data packet till the end 
> of session (this exclude DNS, 3 way handshake, http GET and its ACK).
>              Hystart exit cwnd extracted from nstat (thank you Eric to 
> pointing out to it).
> 
> Overall results:
> 	Download 3MB, Low Bandwidths:
> 		Hystart average download time: 1.41 s
>  		NoHystart average download time: 1.36 s
> 
> 	Download 3MB, High Bandwidths:
> 		Hystart average download time: 0.65 s
>  		NoHystart average download time: 0.37  s
> 
> 	Download 10MB, Low Bandwidths:
> 		Hystart average download time: 2.40 s
>  		NoHystart average download time: 2.12 s
> 
> 	Download 10MB, High Bandwidths:
> 		Hystart average download time: 1.42 s
>  		NoHystart average download time: 0.89  s
> 
> These results show that Hystart has no significant impact in 
> conditions of low available bandwidth, but at high available the 
> decrease of performance can reach 60-70 %.
> With the development of the LTE-A which uses larger spectrum to reach 
> 300 Mbps and more the impact of Hystart will become even more visible.
> 
> I do share with Eric and Neal some files described below and also can 
> provide links to anyone interested.
> 
> Summary of these tests results are gathered in the following Excel 
> files (names, I believe, are self explanatory): Test_3M_low_BW.xlsx, 
> Test_3M_high_BW.xlsx, Test_10M_low_BW.xlsx, Test_10M_high_BW.xlsx.
> Similarly tcpdump traces are called Test_3M_low_BW.pcap, 
> Test_3M_high_BW. pcap, Test_10M_low_BW. pcap, Test_10M_high_BW.
> pcap (able to collect only server side traces, but they are self sufficient).
> For the 10M downloads there are few missing traces as there trace 
> files were recycled. In traces are also ssh sessions, please, ignore 
> them, they were used to put hystart ON and OFF on server and extract 
> nstat results before and after each download.
> 
> Excel files have 2 tabs: curl_sum and Summary.
> In the curl_sum tab the stat is cumulated in chronological order of tests.
> In the Summary_tab  the Hystart and NoHystart tests are shown side by 
> side (as said, Download 1 was with Hystart, Download 2 without, 
> Download 3 with Hystart, Download 4 without, etc.). It also include 
> graphs of Transfer time and a graph which shows the distribution in % 
> of different Hystart exit window DelayCWND.
> 
> Fields:
> comments	- test case
> Transf_Time	- calculated from curl output the duration between first data
> packet and end of transfer
> Transf_speed	- calculated from curl output the ratio of file size and duration
> between first data packet and end of transfer
> Hystart	D_Delay -  hystart exit occurrence (difference between
> TcpExtTCPHystartDelayDetect before and after download)
> D_DelayCwnd	  - Hystart exit cwnd for this download
> (difference of TcpExtTCPHystartDelayCwnd before and after the 
> download) Date_On_Server - time just before the download start, to 
> help navigate the pcap
> 
> Additionally, in the file Test_10M_low_BW.xlsx in the tab curl_sum are 
> included for each download tcp.stream extract from pcap of the first 
> 100 tcp.analysis.ack_rtt which may help in understanding of the 
> hystart behavior (fields ack_rtt_1 to ack_rtt_100).
> In this file there are also fields URI_frame, URI_time, URI_time_relative,
> 	tcp_stream.
> Then there are calculated fields: minRTT as a minimum of all 100 ACK, r1_5,
> r2_8, r3_8,	r4_8 which simulate Hystart calculations of the round curr_rtt
> and r1 min, r2 min, r3 min, r4 min represent minRTT of the whole round.
> Please, correct me if I am wrong, but in my understanding for each ACK 
> the hystart_update is called before the hystart_reset, and as a result 
> the computation of cur_rtt of the 8 packets of the round starts with 
> the 2-nd packet of the round, not the first one.
>  I've use r1_5 as a min(ack_rtt_7...ack_rtt_11), due to 
> hystart_low_window=16, r2_8 as a min(ack_rtt_12 ... ack_rtt_19), r3_8 
> as a
> min(ack_rtt_32 ... ack_rtt_39) and ), r4_8 as a min(ack_rtt_72 ... ack_rtt_79).
> To understand the impact of this there are also r1 HOL, r2 HOL, r3 
> HOL, r4 HOL (Head Of Line) which compute the same with first packet in 
> the round included in the relevant round:  r1_5 as a 
> min(ack_rtt_6...ack_rtt_10), r2_8 as a min(ack_rtt_11 ... ack_rtt_18), 
> r3_8 as a min(ack_rtt_31 ... ack_rtt_38) and ), r4_8 as a min(ack_rtt_71 ... ack_rtt_78).
> 
> If my understanding of Hystart is correct and, assuming in each train 
> the first packet has the lowest RTT which, most probably is valid in 
> all type of networks, then the exit from hystart in this 
> implementation may happen at cwnd 29, 49, 89, 169, etc. (given the IW of 10).
> These tests show that this is the case, however there also 
> intermediate values. However the cwnd of 29 shall be the lowest 
> possible value and this is the case in these tests. Unfortunately, in 
> LTE the first packet in the train has chances to always have ~6 ms 
> lower than the following ones and, because it is not counted in the 
> 2-nd train, there is quite high percentage of exit at 29 (14-36% in 
> these tests), even though later on cubic increases the cwnd without 
> losses up to many hundreds. For me this is too early exit from slow start.
> As it can be seen from comparison of the simulated calculation, 
> inclusion of the first packet in the round curr_rtt calculation would 
> eliminate the cwnd 29 and shift to cwnd 49 or 89 or higher.
> 
> But there are also intermediate values, e.g. 69. Looking through 
> traces one may see that this is due to "compressed ACK" which receives 
> a tightly packed train of many ACK so that in traces there are not 
> data packets sent in turn by server. Looks to me, because 
> hysrart_reset sets the end_seq to snd_next and the server did not 
> managed to send yet packets in reply to this high speed ACK train the 
> snd_next points to much lower value than would normally happen and 
> this actually destroys the correct identification of the borders of 
> the round, the round fails somewhere in the middle of the train not at 
> its border, this makes very likely increased delay due to queueing and 
> earlier exit from slow_start and intermediate values of exit cwnd where they are not expected.
> 
> And last but not least: there are gaps (4-7 ms) in ACK trains. On the 
> perfectly sent IW of 10 data packets train the typical for LTE will be 
> to receive 1 or 2 ACK, then a gap of 4-6 ms then another couple of 
> ACK, then another gap, then 5 ACK compressed together in a fraction of 
> microsecond. Of cause, the next train from the server will contain the 
> same gaps. Very quickly, e.g., the Head of Line (HOL) of the 3-d train 
> will follow immediately the tail of the 2-nd train which will lead to 
> queueing in the end router (actually the radio base station, much 
> earlier than BDP reached), but this will lead to increase of the train 
> RTT and too yearly exit from slow start. Why too early exit ? Because, 
> should server continue increasing the speed this would reduce quicker 
> the gaps both in downlink and uplink, the preice for this being the 
> increase of the RTT (I guess double) against what one would normally see in the end-to-end Ethernet networks.
> 
> 
> I do like ideas of Hystart, but I  don't know how could it be possibly 
> reconciled with my LTE problems.  With the increase of LTE speeds this 
> early exit from slow start will become even more of the problem.
> For LTE, not sure how good and generally acceptable, solution would 
> be, possibly, to increase the HYSTART_DELAY_MIN to 8 ms, which, at 
> least in my case, would diminish the earlier exit significantly, if I 
> believe in these tests results.
> Another solution, which I don't like very much, but as I know some 
> operators use, would be to insert a kind of TCP accelerator between 
> the Internet and the mobile network which will intercept all TCP and 
> would send it to mobile without Hystart but this would kill any 
> end-to-end principle and this will not be the Internet anymore.
> 
> 
> If you have time and never look closely to LTE technology I do a quick 
> summary bellow.
> 
> 
> LTE, TCP self-clocking and ACK compression.
> 
> Spent some time to understand the basics of what could be the reason 
> for all those timing effects due to LTE.
> 	First, unlike wire based transport technologies, LTE (but also UMTS 
> and future 5G) sends data, both uplink and downlink in predefined 
> timeslots called Transmission Time Interval (TTI). In UMTS this is 
> 2ms, in LTE is
> 1 ms and it says that in 5G it will be 0.5 ms.
> These TTI have exact borders and tightly follow each other and require 
> exact synchronization between the mobile device and Radio Base Station 
> (called eNodeB in LTE). To cope with varying radio conditions the 
> sending data bandwidth is modified in such a way that to keep a 
> constant probability of data loss of 10%. Bad radio - less bandwidth, good radio - more bandwidth.
> The bandwidth is modified by packing less or more bits in a TTI of 1 ms in LTE.
> But TTI remain unchanged of exactly 1 ms. That means that, at the 
> TCP/IP layer the precision of the timing information from device to 
> the network is 1 ms. In the case of a bandwidth of 1 Mbps there will 
> be 83 IP packets per second or one packet every 12 ms. In this case, 
> possibly the return of one ACK every 12 ms (every 12 TTI) would be 
> more or less sufficient to keep at a good level the TCP self-clocking 
> mechanism. But, when talking about 100 Mbps (or 300 Mbps) that means 
> one packet (and one ACK) every 0.12 ms and there is no way to provide 
> it with a TTI of 1 ms. This is the reason for ACK compression in LTE 
> and the future 5G which promises more than 1Gbps with its 0.5 ms TTI will be in a big conflict with TCP self-clocking.
> 	But the problem do not stops here. There is a significantly different 
> mechanisms of transmission in the downlink (from eNodeB to mobile 
> device) and in uplink (from mobile device to the eNodeB). The 
> difference come from the fact that the scheduler for both downlink and 
> in uplink is located in the eNodeB. As of my understanding the purpose 
> of this is to simplify to the maximum the device and save its battery. 
> As a result of this design, the eNodeB can send in each TTI toward the 
> mobile device whenever data exist in its buffer (eNodeB is a kind of 
> IP wireless router). But for mobile device, in order to send the data 
> in uplink, after a pause, it has first to ask the network to schedule 
> it. And mobile has for this request (after a pause of sending) just 
> one bit, actually is not even a data bit, is a special radio signal 
> which tells "Hey, I have some data in the buffer to send". Lets look 
> at initial train of IW of 10 packets. In good radio conditions the 
> eNodeB will fit all 10 packets in 1 TTI of 1 ms and send them all to 
> the device. Device will unpack them and generate 10 ACK and then tell 
> to the eNodeB "Hey, I have some data in the buffer to send". The 
> eNodeB has not knowledge how much data the mobile has to send, 
> therefor will allocate some minimum
> capacity: "Well, I give you a grant to send up to 200 Bytes and you 
> must send them exactly after 4 ms you get this notification". These 4 
> ms are allowance for mobile device to prepare data for sending which 
> is a very complicated process, these 4 ms is a must delay between the grant and the transmission.
> If mobile has in its buffer 10 ACK to send it will send only 2 ACK and 
> will complement them with, so called, buffer status report: "I do have 
> still 480 bytes to send". Now network has more information and 
> allocate as
> requested: "Well, I give you a grant to send up to 200 Bytes and you 
> must send them exactly after 4 ms you get this notification". But, in 
> the mean time some other information of higher priority may appear in the device (e.g.
> signaling) and the device will send only 6 out of 8 remaining ACK 
> together with signaling and another Buffer Status Report: "Hey, I do 
> have another 120 bytes to send". And again "Well, I give you a grant 
> to send up to 120 Bytes and you must send them exactly after 4 ms you 
> get this notification". As a result the server will receive 2 ACK, 
> then after a gap of 6 ms another 6 ACK, then, after another 6 ms, the 
> remaining 2 ACK. With the scheduler in the eNodeB these gaps are 
> inevitable. And one more think: mobile pack together those 6 ACK in 1 
> TTI of 1 ms and send them, then the eNodeB unpack them and there is no 
> way to recover the spacing of 0.12 ms between them. eNodeB simply send 
> them to the server back to back at the speed of its data card which is 
> typically 1 Gbps and we face the ACK compression. After the last 2 ACK 
> were sent there is nothing else to send, so there is no Buffer Status 
> Report, therefore, after packets of the next train arrive and ACK are 
> generated everything starts again with one bit of information "Hey, I have some data in the buffer to send".
> That's why there are gaps and ACK compression, loss, or more exactly, 
> lack of any timing information which would be useful for the TCP self-clocking.
> 
> Thank you and sorry for taking your time.
> 
> 
> Veaceslav Roman
> 
> 
> 
> -----Original Message-----
> From: Veaceslav ROMAN
> Sent: Saturday, 05 September 2015 01:10
> To: 'Eric Dumazet'; Neal Cardwell
> Cc: [email-masked]; Piers O'Hanlon; Sangtae Ha; Ingemar Johansson S; 
> Eric Dumazet; [email-masked]
> Subject: RE: [tcpm] [e2e] TCP HyStart patch deployment
> 
> If we look at the first train: all packets received in less than 1 ms. 
> Probably this is only an appearance as LTE transmits in, so called 
> transmission time interval (TTI) of 1 ms, and what we see here is that 
> all 10 packets of initial window fitted in 1 ms, and, when decoded, 
> were presented to the TCP/IP layer (and pcap) all at once. BTW, 8 
> packets of 13022 bytes in 1 ms means instantaneous speed of 104 Mbps, 
> good radio conditions. TCP generates 10 ACK train of the duration of 
> 1.2 ms. Will it be a Fast Ethernet possibly we would consider this normal, isn't it ?
> I looked at how server reply to these 10 ACKs. There are 3 gaps of 4, 
> 2 and 5 ms in the reply train and the total train of 18 (?, it should 
> be 20) packets reaches a duration of 14 ms. AFAIK LTE may introduce 
> gaps in ACK train due to uplink scheduling mechanism.
> May be these gaps trigger Hystart early exit ?
> 
> Veaceslav Roman
> Technical and IT director
> Orange Moldova S.A.
> Fix: +37322575400
> Mob: +37369198400
> Fax: +37322575306
> 
> -----Original Message-----
> From: Eric Dumazet [mailto:[email-masked]]
> Sent: Friday, 04 September 2015 19:48
> To: Neal Cardwell
> Cc: Veaceslav ROMAN; [email-masked]; Piers O'Hanlon; Sangtae Ha; 
> Ingemar Johansson S; Eric Dumazet; [email-masked]
> Subject: Re: [tcpm] [e2e] TCP HyStart patch deployment
> 
> On Fri, 2015-09-04 at 11:32 -0400, Neal Cardwell wrote:
> > Hi Veaceslav,
> >
> > I agree that in your LTE traces it looks like CUBIC Hystart is 
> > exiting slow start too early.
> >
> > SInce you seem to have a nice LTE testbed, would you be able to do 
> > some experiments to find a set of parameters for Hystart that work 
> > better for your LTE environment? For example, you might try the two 
> > variations I suggested earlier in the thread:
> >
> >                         if (ca->curr_rtt > ca->delay_min +
> >                             HYSTART_DELAY_THRESH(ca->delay_min >> 
> > 2)) { or
> >                         if (ca->curr_rtt > ca->delay_min +
> >                             HYSTART_DELAY_THRESH(ca->delay_min >> 
> > 1)) {
> >
> > Do any of those give better results for your tests?
> >
> > neal
> 
> Also, hystart is fooled by too many ACK received in short period.
> 
> This problem would be solved if GRO was used at receiver, as less ACK 
> would be sent.
> 
> Presumably receiver is not a linux TCP stack ?
> 
> Maybe we should add a logic in hystart_update() to take one ACK per ms.
> 
> 05:32:24.134172 IP 94.243.104.156.32924 > 195.95.178.204.80: Flags 
> [S], seq 4294423215, win 65535, options [mss 1460,sackOK,TS val 
> 1112351 ecr 0,nop,wscale 8], length 0
> 05:32:24.161986 IP 195.95.178.204.80 > 94.243.104.156.32924: Flags 
> [S.], seq 1198877721, ack 4294423216, win 28960, options [mss 
> 1416,sackOK,TS val
> 2808808634 ecr 1112351,nop,wscale 7], length 0
> 05:32:24.162108 IP 94.243.104.156.32924 > 195.95.178.204.80: Flags 
> [.], ack 1, win 343, options [nop,nop,TS val 1112354 ecr 2808808634], 
> length 0
> 05:32:24.163072 IP 94.243.104.156.32924 > 195.95.178.204.80: Flags 
> [P.], seq 1:658, ack 1, win 343, options [nop,nop,TS val 1112354 ecr 
> 2808808634], length 657
> 05:32:24.203984 IP 195.95.178.204.80 > 94.243.104.156.32924: Flags 
> [.], ack 658, win 237, options [nop,nop,TS val 2808808675 ecr 
> 1112354], length 0
> 05:32:24.210275 IP 195.95.178.204.80 > 94.243.104.156.32924: Flags 
> [P.], seq 1:387, ack 658, win 237, options [nop,nop,TS val 2808808677 
> ecr 1112354], length 386
> 05:32:24.210315 IP 195.95.178.204.80 > 94.243.104.156.32924: Flags 
> [.], seq 387:1791, ack 658, win 237, options [nop,nop,TS val 
> 2808808677 ecr 1112354], length 1404
> 05:32:24.210332 IP 195.95.178.204.80 > 94.243.104.156.32924: Flags 
> [.], seq 1791:3195, ack 658, win 237, options [nop,nop,TS val 
> 2808808677 ecr 1112354], length 1404
> 05:32:24.210347 IP 195.95.178.204.80 > 94.243.104.156.32924: Flags 
> [.], seq 3195:4599, ack 658, win 237, options [nop,nop,TS val 
> 2808808677 ecr 1112354], length 1404
> 05:32:24.210350 IP 94.243.104.156.32924 > 195.95.178.204.80: Flags 
> [.], ack 387, win 347, options [nop,nop,TS val 1112359 ecr 
> 2808808677], length 0
> 05:32:24.210361 IP 195.95.178.204.80 > 94.243.104.156.32924: Flags 
> [.], seq 4599:6003, ack 658, win 237, options [nop,nop,TS val 
> 2808808677 ecr 1112354], length 1404
> 05:32:24.210374 IP 195.95.178.204.80 > 94.243.104.156.32924: Flags 
> [.], seq 6003:7407, ack 658, win 237, options [nop,nop,TS val 
> 2808808677 ecr 1112354], length 1404
> 05:32:24.210389 IP 195.95.178.204.80 > 94.243.104.156.32924: Flags 
> [.], seq 7407:8811, ack 658, win 237, options [nop,nop,TS val 
> 2808808677 ecr 1112354], length 1404
> 05:32:24.210402 IP 195.95.178.204.80 > 94.243.104.156.32924: Flags 
> [.], seq 8811:10215, ack 658, win 237, options [nop,nop,TS val 
> 2808808677 ecr 1112354], length 1404
> 05:32:24.210416 IP 195.95.178.204.80 > 94.243.104.156.32924: Flags 
> [.], seq 10215:11619, ack 658, win 237, options [nop,nop,TS val 
> 2808808677 ecr 1112354], length 1404
> 05:32:24.210418 IP 94.243.104.156.32924 > 195.95.178.204.80: Flags 
> [.], ack 1791, win 358, options [nop,nop,TS val 1112359 ecr 
> 2808808677], length 0
> 05:32:24.210431 IP 195.95.178.204.80 > 94.243.104.156.32924: Flags 
> [.], seq 11619:13023, ack 658, win 237, options [nop,nop,TS val 
> 2808808677 ecr 1112354], length 1404
> 05:32:24.210455 IP 94.243.104.156.32924 > 195.95.178.204.80: Flags 
> [.], ack 3195, win 369, options [nop,nop,TS val 1112359 ecr 
> 2808808677], length 0
> 05:32:24.210685 IP 94.243.104.156.32924 > 195.95.178.204.80: Flags 
> [.], ack 4599, win 380, options [nop,nop,TS val 1112359 ecr 
> 2808808677], length 0
> 05:32:24.210731 IP 94.243.104.156.32924 > 195.95.178.204.80: Flags 
> [.], ack 6003, win 391, options [nop,nop,TS val 1112359 ecr 
> 2808808677], length 0
> 05:32:24.210937 IP 94.243.104.156.32924 > 195.95.178.204.80: Flags 
> [.], ack 7407, win 402, options [nop,nop,TS val 1112359 ecr 
> 2808808677], length 0
> 05:32:24.211078 IP 94.243.104.156.32924 > 195.95.178.204.80: Flags 
> [.], ack 8811, win 413, options [nop,nop,TS val 1112359 ecr 
> 2808808677], length 0
> 05:32:24.211228 IP 94.243.104.156.32924 > 195.95.178.204.80: Flags 
> [.], ack 10215, win 424, options [nop,nop,TS val 1112359 ecr 
> 2808808677], length 0
> 05:32:24.211377 IP 94.243.104.156.32924 > 195.95.178.204.80: Flags 
> [.], ack 11619, win 435, options [nop,nop,TS val 1112359 ecr 
> 2808808677], length 0
> 05:32:24.211544 IP 94.243.104.156.32924 > 195.95.178.204.80: Flags 
> [.], ack 13023, win 446, options [nop,nop,TS val 1112359 ecr 
> 2808808677], length 0
> 05:32:24.237999 IP 195.95.178.204.80 > 94.243.104.156.32924: Flags 
> [.], seq 13023:14427, ack 658, win 237, options [nop,nop,TS val 
> 2808808709 ecr 1112359], length 1404
> 05:32:24.238053 IP 195.95.178.204.80 > 94.243.104.156.32924: Flags 
> [.], seq 14427:15831, ack 658, win 237, options [nop,nop,TS val 
> 2808808709 ecr 1112359], length 1404
> 05:32:24.238090 IP 94.243.104.156.32924 > 195.95.178.204.80: Flags 
> [.], ack 14427, win 457, options [nop,nop,TS val 1112362 ecr 
> 2808808709], length 0
> 05:32:24.238164 IP 94.243.104.156.32924 > 195.95.178.204.80: Flags 
> [.], ack 15831, win 468, options [nop,nop,TS val 1112362 ecr 
> 2808808709], length 0
> 05:32:24.244024 IP 195.95.178.204.80 > 94.243.104.156.32924: Flags 
> [.], seq 15831:17235, ack 658, win 237, options [nop,nop,TS val 
> 2808808713 ecr 1112359], length 1404
> 05:32:24.244063 IP 195.95.178.204.80 > 94.243.104.156.32924: Flags 
> [.], seq 17235:18639, ack 658, win 237, options [nop,nop,TS val 
> 2808808713 ecr 1112359], length 1404
>

---

From: Ingemar Johansson S <[email-masked]>
Date: Thu, 1 Oct 2015 09:59:38 +0000

Hi

Yes, the variable delay_min is in Q3 for reasons of precision I guess
But the call to HYSTART_DELAY_THRESH is with delay_min shifted down to Q0

HYSTART_DELAY_THRESH(ca->delay_min >> 3))  (see http://lxr.free-electrons.com/source/net/ipv4/tcp_cubic.c#L403 )
I may be off here.. 

Regarding pacing. I don't have real data to show the effects of pacing, in simulations it is however possible to see the gains in terms of less coalescing, yes pacing does not solve the ACK compression but it seem to solve the follow up effect that ACK compression gives.

/Ingemar

> Sent: den 1 oktober 2015 10:32
> To: Ingemar Johansson S; Eric Dumazet; Neal Cardwell
> Cc: [email-masked]; Piers O'Hanlon; Sangtae Ha; Eric Dumazet; end2end-
> [email-masked]
> 		delay = (rtt_us << 3) / USEC_PER_MSEC; So,
> 4U<<3 is equivalent to 4 ms .
> 
> Regarding pacing I am not sure will help the performance: there are gaps in
> ACK trains of the order of milliseconds and this is a lost time and there is no
> way to compensate it. If introduce ACK pacing or data packets pacing this
> would add to the overall lost time.
> Pacing can help only to alleviate a potential problem of some downstream
> routers with insufficient buffers which may introduce unexpected packets
> drops when large train sent at a high speed as a reply to highly compressed
> ACK train, but, on another hands.
> But how much pacing and how to accommodate to largely changing radio
> conditions ? And with permanently changing technology and speed ? Anyway
> the pacing will add to the problem of gaps and better size properly routers
> buffers.
> 
> Ideally ACK train is split in 2 parts, but in practice I do see more. But as the
> stream progress the less gaps. The bigger the downlink pressure the more
> ACK and less chances to have a pause in ACK sending, then there will be a
> continuous flow of Buffer Status Reports and less of the Scheduling Request
> (SR, "Hey, I have some data to send"). The only issue I am not sure is on the
> potential gaps in other networks. I did not put bellow all details, but you
> should know that device can not send SR in arbitrary moments of time but in
> predefined for him intervals. In my network this period is 5 ms, therefore the
> RTT of the first ACK in the train will have variations of 2.5 ms. But standards
> admit this period to be also 10 ms ,20 ms, 40 ms, 80 ms and I can not say what
> is the practice of others. I may only guess that most will try to use the best
> and fastest, i.e. 5 ms.
> Anyway, to get a realistic modeling of LTE interaction with TCP the detailed
> MAC scheduler model is needed. Unfortunately, most of the literature focus
> on scheduling of the Physical Resource Blocks, Adaptive Modulation and
> Coding, MIMO, i.e., focusing on the distribution of resources between users,
> which is good and needed, but do not address very much the timing
> interaction with TCP.  At least, I never met in LTE scheduling literature a term
> "TCP self-clocking".
> 
> The 8 ms  HYSTART_DELAY_MIN could only be a simple solution for the
> problem of counting of the round' curr_rtt from the second ACK of the train
> (as the second ACK RTT will be, typically, due to scheduling, 5-7 ms longer
> than the first one), but this only if SR periodicity is 5 ms.
> Thanks for the detailed report. What seems unclear to me is the value of
> HYSTART_DELAY_MIN. You mention 8ms but when I look into the 4.2 kernel
> code I get the feeling that it is 32ms ( #define HYSTART_DELAY_MIN
> (4U<<3)   ). Is this a mistake from my side  ?
> I have run LTE simulations and tried to wring this HyStart issue inside out and
> upside down but sofar it has been very difficult to make HyStart exit
> prematurely but then of course I have run with HYSTART_DELAY_MIN =
> 32ms.
> 
> As regards to the ACK-compression problem, yes, ACK compression easily
> occur in LTE, even in simulations. What I have seen is that packet pacing is a
> very efficient remedy, for instance the packet implemented in QUIC seems
> to solve ACK compression issues very well.
> 
> It is interesting what you say that ACK trains are so much split up, I could
> understand that they are split up in two parts (initial grant + additional after
> received BSR).
> 
> /Ingemar
> 
> > and the future 5G which promises more than 1Gbps with its 0.5 ms TTI will
> be in a big conflict with TCP self-clocking.

---

From: Veaceslav ROMAN <[email-masked]>
Date: Thu, 1 Oct 2015 11:45:53 +0000

Well this say 
	If curr_rtt is higher than delay_min + (1/8 * delay_min or 4 ms, whichever is higher)  then exit slow start.

When actual minimum delay is less than 32 ms the delay_min will be less than 32<<3, then the delay_min>>3 will be less than 32 and will compare with 4<<3 which is 32 and therefor the threshold will be (real_curr_rtt<<3) > (real_delay<<3 + 4<<3) or (real_curr_rtt)>(real_delay +4)

Veaceslav Roman

-----Original Message-----
From: Ingemar Johansson S [mailto:[email-masked]] 
Sent: Thursday, 01 October 2015 13:00
To: Veaceslav ROMAN; Eric Dumazet; Neal Cardwell
Cc: [email-masked]; Piers O'Hanlon; Sangtae Ha; Eric Dumazet; [email-masked]; Ingemar Johansson S
Subject: RE: [tcpm] [e2e] TCP HyStart patch deployment

Hi

Yes, the variable delay_min is in Q3 for reasons of precision I guess But the call to HYSTART_DELAY_THRESH is with delay_min shifted down to Q0

HYSTART_DELAY_THRESH(ca->delay_min >> 3))  (see http://lxr.free-electrons.com/source/net/ipv4/tcp_cubic.c#L403 ) I may be off here.. 

/Ingemar

> Regarding pacing I am not sure will help the performance: there are 
> gaps in ACK trains of the order of milliseconds and this is a lost 
> time and there is no way to compensate it. If introduce ACK pacing or 
> data packets pacing this would add to the overall lost time.
> Pacing can help only to alleviate a potential problem of some 
> downstream routers with insufficient buffers which may introduce 
> unexpected packets drops when large train sent at a high speed as a 
> reply to highly compressed ACK train, but, on another hands.
> conditions ? And with permanently changing technology and speed ? 
> Anyway the pacing will add to the problem of gaps and better size 
> properly routers buffers.
> 
> Ideally ACK train is split in 2 parts, but in practice I do see more. 
> But as the stream progress the less gaps. The bigger the downlink 
> pressure the more ACK and less chances to have a pause in ACK sending, 
> then there will be a continuous flow of Buffer Status Reports and less 
> of the Scheduling Request (SR, "Hey, I have some data to send"). The 
> only issue I am not sure is on the potential gaps in other networks. I 
> did not put bellow all details, but you should know that device can 
> not send SR in arbitrary moments of time but in predefined for him 
> intervals. In my network this period is 5 ms, therefore the RTT of the 
> first ACK in the train will have variations of 2.5 ms. But standards 
> admit this period to be also 10 ms ,20 ms, 40 ms, 80 ms and I can not 
> say what is the practice of others. I may only guess that most will try to use the best and fastest, i.e. 5 ms.
> Anyway, to get a realistic modeling of LTE interaction with TCP the 
> detailed MAC scheduler model is needed. Unfortunately, most of the 
> literature focus on scheduling of the Physical Resource Blocks, 
> Adaptive Modulation and Coding, MIMO, i.e., focusing on the 
> distribution of resources between users, which is good and needed, but 
> do not address very much the timing interaction with TCP.  At least, I 
> never met in LTE scheduling literature a term "TCP self-clocking".
> 
> problem of counting of the round' curr_rtt from the second ACK of the 
> train (as the second ACK RTT will be, typically, due to scheduling, 
> 5-7 ms longer than the first one), but this only if SR periodicity is 5 ms.
> [email-masked]; Ingemar Johansson S
> Thanks for the detailed report. What seems unclear to me is the value 
> of HYSTART_DELAY_MIN. You mention 8ms but when I look into the 4.2 
> kernel code I get the feeling that it is 32ms ( #define HYSTART_DELAY_MIN
> I have run LTE simulations and tried to wring this HyStart issue 
> inside out and upside down but sofar it has been very difficult to 
> make HyStart exit prematurely but then of course I have run with 
> HYSTART_DELAY_MIN = 32ms.
> 
> occur in LTE, even in simulations. What I have seen is that packet 
> pacing is a very efficient remedy, for instance the packet implemented 
> in QUIC seems to solve ACK compression issues very well.
> 
> It is interesting what you say that ACK trains are so much split up, I 
> could understand that they are split up in two parts (initial grant + 
> additional after received BSR).
> 
> /Ingemar
> 
> > side as shown by curl from the first received data packet till the 
> > end of session (this exclude DNS, 3 way handshake, http GET and its ACK).
> >              Hystart exit cwnd extracted from nstat (thank you Eric 
> > to pointing out to it).
> >
> > Overall results:
> > I do share with Eric and Neal some files described below and also 
> > can provide links to anyone interested.
> >
> > In the Summary_tab  the Hystart and NoHystart tests are shown side 
> > by side (as said, Download 1 was with Hystart, Download 2 without, 
> > graphs of Transfer time and a graph which shows the distribution in 
> > % of different Hystart exit window DelayCWND.
> >
> > Fields:
> > comments	- test case
> > Additionally, in the file Test_10M_low_BW.xlsx in the tab curl_sum 
> > are included for each download tcp.stream extract from pcap of the 
> > first
> > Please, correct me if I am wrong, but in my understanding for each 
> > ACK the hystart_update is called before the hystart_reset, and as a 
> > result the computation of cur_rtt of the 8 packets of the round 
> > starts with the 2-nd packet of the round, not the first one.
> > hystart_low_window=16, r2_8 as a min(ack_rtt_12 ... ack_rtt_19), 
> > r3_8 as a
> > HOL, r4 HOL (Head Of Line) which compute the same with first packet 
> > in the round included in the relevant round:  r1_5 as a 
> > If my understanding of Hystart is correct and, assuming in each 
> > train the first packet has the lowest RTT which, most probably is 
> > valid in all type of networks, then the exit from hystart in this 
> > implementation may happen at cwnd 29, 49, 89, 169, etc. (given the 
> > IW of
> 10).
> > possible value and this is the case in these tests. Unfortunately, 
> > in LTE the first packet in the train has chances to always have ~6 
> > ms lower than the following ones and, because it is not counted in 
> > the 2-nd train, there is quite high percentage of exit at 29 (14-36% 
> > in these tests), even though later on cubic increases the cwnd 
> > without losses up to many hundreds. For me this is too early exit from slow start.
> > inclusion of the first packet in the round curr_rtt calculation 
> > would eliminate the cwnd 29 and shift to cwnd 49 or 89 or higher.
> >
> > traces one may see that this is due to "compressed ACK" which 
> > receives a tightly packed train of many ACK so that in traces there 
> > are not data packets sent in turn by server. Looks to me, because 
> > managed to send yet packets in reply to this high speed ACK train 
> > the snd_next points to much lower value than would normally happen 
> > and this actually destroys the correct identification of the borders 
> > of the round, the round fails somewhere in the middle of the train 
> > not at its border, this makes very likely increased delay due to 
> > queueing and earlier exit from slow_start and intermediate values of 
> > exit cwnd where
> they are not expected.
> >
> > And last but not least: there are gaps (4-7 ms) in ACK trains. On 
> > the perfectly sent IW of 10 data packets train the typical for LTE 
> > will be to receive 1 or 2 ACK, then a gap of 4-6 ms then another 
> > couple of ACK, then another gap, then 5 ACK compressed together in a 
> > fraction of microsecond. Of cause, the next train from the server 
> > will contain the same gaps. Very quickly, e.g., the Head of Line 
> > (HOL) of the 3-d train will follow immediately the tail of the 2-nd 
> > train which will lead to queueing in the end router (actually the 
> > radio base station, much earlier than BDP reached), but this will 
> > lead to increase of the train RTT and too yearly exit from slow 
> > start. Why too early exit ? Because, should server continue 
> > increasing the speed this would reduce quicker the gaps both in 
> > downlink and uplink, the preice for this being the increase of the 
> > RTT (I guess double) against what one would normally see
> in the end-to-end Ethernet networks.
> >
> >
> > I do like ideas of Hystart, but I  don't know how could it be 
> > possibly reconciled with my LTE problems.  With the increase of LTE 
> > speeds this early exit from slow start will become even more of the problem.
> > least in my case, would diminish the earlier exit significantly, if 
> > I believe in these tests results.
> > If you have time and never look closely to LTE technology I do a 
> > quick summary bellow.
> >
> >
> > These TTI have exact borders and tightly follow each other and 
> > require exact synchronization between the mobile device and Radio 
> > Base Station (called eNodeB in LTE). To cope with varying radio 
> > conditions the sending data bandwidth is modified in such a way that 
> > to keep a constant probability of data loss of 10%. Bad radio - less 
> > bandwidth, good
> radio - more bandwidth.
> > The bandwidth is modified by packing less or more bits in a TTI of 1 
> > ms in
> LTE.
> > more or less sufficient to keep at a good level the TCP 
> > self-clocking mechanism. But, when talking about 100 Mbps (or 300 
> > Mbps) that means one packet (and one ACK) every 0.12 ms and there is 
> > no way to provide it with a TTI of 1 ms. This is the reason for ACK 
> > compression in LTE and the future 5G which promises more than 1Gbps 
> > with its 0.5 ms TTI will
> > difference come from the fact that the scheduler for both downlink 
> > and in uplink is located in the eNodeB. As of my understanding the 
> > purpose of this is to simplify to the maximum the device and save its battery.
> > As a result of this design, the eNodeB can send in each TTI toward 
> > the mobile device whenever data exist in its buffer (eNodeB is a 
> > kind of IP wireless router). But for mobile device, in order to send 
> > the data in uplink, after a pause, it has first to ask the network 
> > to schedule it. And mobile has for this request (after a pause of 
> > sending) just one bit, actually is not even a data bit, is a special 
> > radio signal which tells "Hey, I have some data in the buffer to 
> > send". Lets look at initial train of IW of 10 packets. In good radio 
> > conditions the eNodeB will fit all 10 packets in 1 TTI of 1 ms and 
> > send them all to the device. Device will unpack them and generate 10 
> > ACK and then tell to the eNodeB "Hey, I have some data in the buffer 
> > to send". The eNodeB has not knowledge how much data the mobile has 
> > to send, therefor will allocate some minimum
> > must send them exactly after 4 ms you get this notification". These 
> > 4 ms are allowance for mobile device to prepare data for sending 
> > which is a very complicated process, these 4 ms is a must delay 
> > between the grant
> and the transmission.
> > If mobile has in its buffer 10 ACK to send it will send only 2 ACK 
> > and will complement them with, so called, buffer status report: "I 
> > do have still 480 bytes to send". Now network has more information 
> > and allocate as
> > must send them exactly after 4 ms you get this notification". But, 
> > in the mean time some other information of higher priority may 
> > appear in the
> device (e.g.
> > to send up to 120 Bytes and you must send them exactly after 4 ms 
> > you get this notification". As a result the server will receive 2 
> > ACK, then after a gap of 6 ms another 6 ACK, then, after another 6 
> > ms, the remaining 2 ACK. With the scheduler in the eNodeB these gaps 
> > are inevitable. And one more think: mobile pack together those 6 ACK 
> > in 1 TTI of 1 ms and send them, then the eNodeB unpack them and 
> > there is no way to recover the spacing of 0.12 ms between them. 
> > eNodeB simply send them to the server back to back at the speed of 
> > its data card which is typically 1 Gbps and we face the ACK 
> > compression. After the last 2 ACK were sent there is nothing else to 
> > send, so there is no Buffer Status Report, therefore, after packets 
> > of the next train arrive and ACK are generated everything starts 
> > again with one bit of information "Hey, I have
> some data in the buffer to send".
> > That's why there are gaps and ACK compression, loss, or more 
> > exactly, lack of any timing information which would be useful for 
> > the TCP self-
> clocking.
> >
> > transmission time interval (TTI) of 1 ms, and what we see here is 
> > that all 10 packets of initial window fitted in 1 ms, and, when 
> > decoded, were presented to the TCP/IP layer (and pcap) all at once. 
> > BTW, 8 packets of 13022 bytes in 1 ms means instantaneous speed of 
> > 104 Mbps, good radio conditions. TCP generates 10 ACK train of the 
> > duration of
> > 1.2 ms. Will it be a Fast Ethernet possibly we would consider this 
> > normal,
> isn't it ?
> > 2 and 5 ms in the reply train and the total train of 18 (?, it 
> > should be 20) packets reaches a duration of 14 ms. AFAIK LTE may 
> > introduce gaps in ACK train due to uplink scheduling mechanism.
> > > better for your LTE environment? For example, you might try the 
> > > two variations I suggested earlier in the thread:
> > >
> > This problem would be solved if GRO was used at receiver, as less 
> > ACK would be sent.
> >
> > [.], ack 1, win 343, options [nop,nop,TS val 1112354 ecr 
> > 2808808634], length 0
> > [P.], seq 1:387, ack 658, win 237, options [nop,nop,TS val 
> > 2808808677 ecr 1112354], length 386

---

From: Ingemar Johansson S <[email-masked]>
Date: Thu, 1 Oct 2015 11:57:40 +0000

Hmm 
You are right !. 
Well... That of course should explain why HyStart performs badly for LTE... Atleast it is part of the reason.

/Ingemar

> Sent: den 1 oktober 2015 13:46
> 	If curr_rtt is higher than delay_min + (1/8 * delay_min or 4 ms,
> whichever is higher)  then exit slow start.
> 
> When actual minimum delay is less than 32 ms the delay_min will be less than
> 32<<3, then the delay_min>>3 will be less than 32 and will compare with
> 4<<3 which is 32 and therefor the threshold will be (real_curr_rtt<<3) >
> (real_delay<<3 + 4<<3) or (real_curr_rtt)>(real_delay +4)
> 
> 
> Veaceslav Roman
> 
> Yes, the variable delay_min is in Q3 for reasons of precision I guess But the
> call to HYSTART_DELAY_THRESH is with delay_min shifted down to Q0
> 
> HYSTART_DELAY_THRESH(ca->delay_min >> 3))  (see http://lxr.free-
> electrons.com/source/net/ipv4/tcp_cubic.c#L403 ) I may be off here..
> 
> Regarding pacing. I don't have real data to show the effects of pacing, in
> simulations it is however possible to see the gains in terms of less coalescing,
> yes pacing does not solve the ACK compression but it seem to solve the
> follow up effect that ACK compression gives.
> 
> /Ingemar
>

---

From: Veaceslav ROMAN <[email-masked]>
Date: Thu, 1 Oct 2015 15:17:06 +0000

Sure 10 ms will be better as more of the LTE technological jumps will be covered and, at least, will help get rid of exit at the minimum exit cwnd (29, which at 20 ms RTT is ~16Mbps) and will shift to 49 (~28 Mbps) or to 89 (~ 50 Mbps).  
We have all necessary to repeat those tests.  

Veaceslav Roman

-----Original Message-----
From: Eric Dumazet [mailto:[email-masked]] 
Sent: Thursday, 01 October 2015 15:44
To: Ingemar Johansson S
Cc: Veaceslav ROMAN; Eric Dumazet; Neal Cardwell; [email-masked]; Piers O'Hanlon; Sangtae Ha; [email-masked]
Subject: Re: [tcpm] [e2e] TCP HyStart patch deployment

Yes, this 4ms value problem is even more visible with TCP pacing (as implemented with linux sch_fq packet scheduler) because of apparent delays caused by sojourn time in packet scheduler itself.
TCP does not offset this sojourn time (as RTT is also used for RTO determination, better have an inflated view)

Is switching to 10 ms would be better in your LTE tests ?

On Thu, Oct 1, 2015 at 4:57 AM, Ingemar Johansson S <[email-masked]> wrote:
>>       If curr_rtt is higher than delay_min + (1/8 * delay_min or 4 
>> ms, whichever is higher)  then exit slow start.
>>
>> When actual minimum delay is less than 32 ms the delay_min will be 
>> less than 32<<3, then the delay_min>>3 will be less than 32 and will 
>> compare with
>> Regarding pacing. I don't have real data to show the effects of 
>> pacing, in simulations it is however possible to see the gains in 
>> terms of less coalescing, yes pacing does not solve the ACK 
>> compression but it seem to solve the follow up effect that ACK compression gives.
>>
>> /Ingemar
>>
>> > Cc: [email-masked]; Piers O'Hanlon; Sangtae Ha; Eric Dumazet; 
>> > end2end- [email-masked]
>> > time and there is no way to compensate it. If introduce ACK pacing 
>> > or data packets pacing this would add to the overall lost time.
>> > But how much pacing and how to accommodate to largely changing 
>> > radio conditions ? And with permanently changing technology and speed ?
>> > pressure the more ACK and less chances to have a pause in ACK 
>> > sending, then there will be a continuous flow of Buffer Status 
>> > Reports and less of the Scheduling Request (SR, "Hey, I have some 
>> > data to send"). The only issue I am not sure is on the potential 
>> > gaps in other networks. I did not put bellow all details, but you 
>> > should know that device can not send SR in arbitrary moments of 
>> > time but in predefined for him intervals. In my network this period 
>> > is 5 ms, therefore the RTT of the first ACK in the train will have 
>> > variations of 2.5 ms. But standards admit this period to be also 10 
>> > ms ,20 ms, 40 ms, 80 ms and I can not say what is the practice of 
>> > others. I may only guess that most will try to use
>> the best and fastest, i.e. 5 ms.
>> > distribution of resources between users, which is good and needed, 
>> > but do not address very much the timing interaction with TCP.  At 
>> > least, I never met in LTE scheduling literature a term "TCP self-clocking".
>> >
>> > problem of counting of the round' curr_rtt from the second ACK of 
>> > the train (as the second ACK RTT will be, typically, due to 
>> > scheduling,
>> > Cc: [email-masked]; Piers O'Hanlon; Sangtae Ha; Eric Dumazet; 
>> > end2end- [email-masked]; Ingemar Johansson S
>> > Thanks for the detailed report. What seems unclear to me is the 
>> > value of HYSTART_DELAY_MIN. You mention 8ms but when I look into 
>> > the 4.2 kernel code I get the feeling that it is 32ms ( #define HYSTART_DELAY_MIN
>> > As regards to the ACK-compression problem, yes, ACK compression 
>> > easily occur in LTE, even in simulations. What I have seen is that 
>> > packet pacing is a very efficient remedy, for instance the packet 
>> > implemented in QUIC seems to solve ACK compression issues very well.
>> >
>> > It is interesting what you say that ACK trains are so much split 
>> > up, I could understand that they are split up in two parts (initial 
>> > grant + additional after received BSR).
>> >
>> > /Ingemar
>> >
>> > > Cc: [email-masked]; Piers O'Hanlon; Sangtae Ha; Ingemar Johansson 
>> > > S; Eric Dumazet; [email-masked]
>> > >              Distance between the Core Network (EPC) and Base 
>> > > Station - 10
>> km
>> > >             Server: Linux k40srv 4.1.0-040100-generic 
>> > > #201507030940 SMP Fri Jul 3
>> > >              Download duration and bandwidth measured at the 
>> > > client side as shown by curl from the first received data packet 
>> > > till the end of session (this exclude DNS, 3 way handshake, http 
>> > > GET and its
>> ACK).
>> > >              Hystart exit cwnd extracted from nstat (thank you 
>> > > Eric to pointing out to it).
>> > >
>> > > Overall results:
>> > > Summary of these tests results are gathered in the following 
>> > > Excel files (names, I believe, are self explanatory): 
>> > > Test_3M_low_BW.xlsx, Test_3M_high_BW.xlsx, Test_10M_low_BW.xlsx,
>> > Test_10M_high_BW.xlsx.
>> > > files were recycled. In traces are also ssh sessions, please, 
>> > > ignore them, they were used to put hystart ON and OFF on server 
>> > > and extract nstat results before and after each download.
>> > >
>> > > In the Summary_tab  the Hystart and NoHystart tests are shown 
>> > > side by side (as said, Download 1 was with Hystart, Download 2 
>> > > without, Download 3 with Hystart, Download 4 without, etc.). It 
>> > > also include graphs of Transfer time and a graph which shows the 
>> > > distribution in % of different Hystart exit window DelayCWND.
>> > >
>> > > Fields:
>> > > comments  - test case
>> > > download) Date_On_Server - time just before the download start, 
>> > > to help navigate the pcap
>> > >
>> > > Additionally, in the file Test_10M_low_BW.xlsx in the tab 
>> > > curl_sum are included for each download tcp.stream extract from 
>> > > pcap of the first
>> > > Then there are calculated fields: minRTT as a minimum of all 100 
>> > > ACK,
>> r1_5,
>> > > Please, correct me if I am wrong, but in my understanding for 
>> > > each ACK the hystart_update is called before the hystart_reset, 
>> > > and as a result the computation of cur_rtt of the 8 packets of 
>> > > the round starts with the 2-nd packet of the round, not the first one.
>> > > To understand the impact of this there are also r1 HOL, r2 HOL, 
>> > > r3 HOL, r4 HOL (Head Of Line) which compute the same with first 
>> > > packet in the round included in the relevant round:  r1_5 as a 
>> > > implementation may happen at cwnd 29, 49, 89, 169, etc. (given 
>> > > the IW of
>> > 10).
>> > > possible value and this is the case in these tests. 
>> > > Unfortunately, in LTE the first packet in the train has chances 
>> > > to always have ~6 ms lower than the following ones and, because 
>> > > it is not counted in the 2-nd train, there is quite high 
>> > > percentage of exit at 29 (14-36% in these tests), even though 
>> > > later on cubic increases the cwnd without losses up to many 
>> > > hundreds. For me this is too early exit from
>> slow start.
>> > > receives a tightly packed train of many ACK so that in traces 
>> > > there are not data packets sent in turn by server. Looks to me, 
>> > > because hysrart_reset sets the end_seq to snd_next and the server 
>> > > did not managed to send yet packets in reply to this high speed 
>> > > ACK train the snd_next points to much lower value than would 
>> > > normally happen and this actually destroys the correct 
>> > > identification of the borders of the round, the round fails 
>> > > somewhere in the middle of the train not at its border, this 
>> > > makes very likely increased delay due to queueing and earlier 
>> > > exit from slow_start and intermediate values of exit cwnd where
>> > they are not expected.
>> > >
>> > > the perfectly sent IW of 10 data packets train the typical for 
>> > > LTE will be to receive 1 or 2 ACK, then a gap of 4-6 ms then 
>> > > another couple of ACK, then another gap, then 5 ACK compressed 
>> > > together in a fraction of microsecond. Of cause, the next train 
>> > > from the server will contain the same gaps. Very quickly, e.g., 
>> > > the Head of Line
>> > > (HOL) of the 3-d train will follow immediately the tail of the 
>> > > 2-nd train which will lead to queueing in the end router 
>> > > (actually the radio base station, much earlier than BDP reached), 
>> > > but this will lead to increase of the train RTT and too yearly 
>> > > exit from slow start. Why too early exit ? Because, should server 
>> > > continue increasing the speed this would reduce quicker the gaps 
>> > > both in downlink and uplink, the preice for this being the 
>> > > increase of the RTT (I guess double) against what one would 
>> > > normally see
>> > > possibly reconciled with my LTE problems.  With the increase of 
>> > > LTE speeds this early exit from slow start will become even more 
>> > > of the
>> problem.
>> > > For LTE, not sure how good and generally acceptable, solution 
>> > > would be, possibly, to increase the HYSTART_DELAY_MIN to 8 ms, 
>> > > which, at least in my case, would diminish the earlier exit 
>> > > significantly, if I believe in these tests results.
>> > > Another solution, which I don't like very much, but as I know 
>> > > some operators use, would be to insert a kind of TCP accelerator 
>> > > between the Internet and the mobile network which will intercept 
>> > > all TCP and would send it to mobile without Hystart but this 
>> > > would kill any end-to-end principle and this will not be the Internet anymore.
>> > >
>> > >
>> > > Spent some time to understand the basics of what could be the 
>> > > reason for all those timing effects due to LTE.
>> > > timeslots called Transmission Time Interval (TTI). In UMTS this 
>> > > is 2ms, in LTE is
>> > > conditions the sending data bandwidth is modified in such a way 
>> > > that to keep a constant probability of data loss of 10%. Bad 
>> > > radio - less bandwidth, good
>> > radio - more bandwidth.
>> > > The bandwidth is modified by packing less or more bits in a TTI 
>> > > of 1 ms in
>> > LTE.
>> > > TCP/IP layer the precision of the timing information from device 
>> > > to the network is 1 ms. In the case of a bandwidth of 1 Mbps 
>> > > there will be 83 IP packets per second or one packet every 12 ms. 
>> > > In this case, possibly the return of one ACK every 12 ms (every 
>> > > 12 TTI) would be more or less sufficient to keep at a good level 
>> > > the TCP self-clocking mechanism. But, when talking about 100 Mbps 
>> > > (or 300
>> > > Mbps) that means one packet (and one ACK) every 0.12 ms and there 
>> > > is no way to provide it with a TTI of 1 ms. This is the reason 
>> > > for ACK compression in LTE and the future 5G which promises more 
>> > > than 1Gbps with its 0.5 ms TTI will
>> > > difference come from the fact that the scheduler for both 
>> > > downlink and in uplink is located in the eNodeB. As of my 
>> > > understanding the purpose of this is to simplify to the maximum 
>> > > the device and save its
>> battery.
>> > > As a result of this design, the eNodeB can send in each TTI 
>> > > toward the mobile device whenever data exist in its buffer 
>> > > (eNodeB is a kind of IP wireless router). But for mobile device, 
>> > > in order to send the data in uplink, after a pause, it has first 
>> > > to ask the network to schedule it. And mobile has for this 
>> > > request (after a pause of
>> > > sending) just one bit, actually is not even a data bit, is a 
>> > > special radio signal which tells "Hey, I have some data in the 
>> > > buffer to send". Lets look at initial train of IW of 10 packets. 
>> > > In good radio conditions the eNodeB will fit all 10 packets in 1 
>> > > TTI of 1 ms and send them all to the device. Device will unpack 
>> > > them and generate 10 ACK and then tell to the eNodeB "Hey, I have 
>> > > some data in the buffer to send". The eNodeB has not knowledge 
>> > > how much data the mobile has to send, therefor will allocate some 
>> > > minimum
>> > > capacity: "Well, I give you a grant to send up to 200 Bytes and 
>> > > you must send them exactly after 4 ms you get this notification". 
>> > > These
>> > > If mobile has in its buffer 10 ACK to send it will send only 2 
>> > > ACK and will complement them with, so called, buffer status 
>> > > report: "I do have still 480 bytes to send". Now network has more 
>> > > information and allocate as
>> > > requested: "Well, I give you a grant to send up to 200 Bytes and 
>> > > you must send them exactly after 4 ms you get this notification". 
>> > > But, in the mean time some other information of higher priority 
>> > > may appear in the
>> > device (e.g.
>> > > together with signaling and another Buffer Status Report: "Hey, I 
>> > > do have another 120 bytes to send". And again "Well, I give you a 
>> > > grant to send up to 120 Bytes and you must send them exactly 
>> > > after 4 ms you get this notification". As a result the server 
>> > > will receive 2 ACK, then after a gap of 6 ms another 6 ACK, then, 
>> > > after another 6 ms, the remaining 2 ACK. With the scheduler in 
>> > > the eNodeB these gaps are inevitable. And one more think: mobile 
>> > > pack together those 6 ACK in 1 TTI of 1 ms and send them, then 
>> > > the eNodeB unpack them and there is no way to recover the spacing of 0.12 ms between them.
>> > > eNodeB simply send them to the server back to back at the speed 
>> > > of its data card which is typically 1 Gbps and we face the ACK 
>> > > compression. After the last 2 ACK were sent there is nothing else 
>> > > to send, so there is no Buffer Status Report, therefore, after 
>> > > packets of the next train arrive and ACK are generated everything 
>> > > starts again with one bit of information "Hey, I have
>> > > Cc: [email-masked]; Piers O'Hanlon; Sangtae Ha; Ingemar Johansson 
>> > > S; Eric Dumazet; [email-masked]
>> > > Probably this is only an appearance as LTE transmits in, so 
>> > > called transmission time interval (TTI) of 1 ms, and what we see 
>> > > here is that all 10 packets of initial window fitted in 1 ms, 
>> > > and, when decoded, were presented to the TCP/IP layer (and pcap) all at once.
>> > > 104 Mbps, good radio conditions. TCP generates 10 ACK train of 
>> > > the duration of
>> > > 1.2 ms. Will it be a Fast Ethernet possibly we would consider 
>> > > this normal,
>> > isn't it ?
>> > > > SInce you seem to have a nice LTE testbed, would you be able to 
>> > > > do some experiments to find a set of parameters for Hystart 
>> > > > that work better for your LTE environment? For example, you 
>> > > > might try the two variations I suggested earlier in the thread:
>> > > >
>> > > >                             HYSTART_DELAY_THRESH(ca->delay_min 
>> > > > >>
>> > > > 2)) { or
>> > > >                             HYSTART_DELAY_THRESH(ca->delay_min 
>> > > > >>
>> > > > 1)) {
>> > > >
>> > > 05:32:24.134172 IP 94.243.104.156.32924 > 195.95.178.204.80: 
>> > > Flags [S], seq 4294423215, win 65535, options [mss 1460,sackOK,TS 
>> > > val
>> > > 05:32:24.161986 IP 195.95.178.204.80 > 94.243.104.156.32924: 
>> > > Flags [S.], seq 1198877721, ack 4294423216, win 28960, options 
>> > > [mss 1416,sackOK,TS val
>> > > 05:32:24.162108 IP 94.243.104.156.32924 > 195.95.178.204.80: 
>> > > Flags [.], ack 1, win 343, options [nop,nop,TS val 1112354 ecr 
>> > > 2808808634], length 0
>> > > 05:32:24.163072 IP 94.243.104.156.32924 > 195.95.178.204.80: 
>> > > Flags [P.], seq 1:658, ack 1, win 343, options [nop,nop,TS val 
>> > > 1112354 ecr 2808808634], length 657
>> > > 05:32:24.203984 IP 195.95.178.204.80 > 94.243.104.156.32924: 
>> > > Flags [.], ack 658, win 237, options [nop,nop,TS val 2808808675 
>> > > ecr 1112354], length 0
>> > > 05:32:24.210275 IP 195.95.178.204.80 > 94.243.104.156.32924: 
>> > > Flags [P.], seq 1:387, ack 658, win 237, options [nop,nop,TS val
>> > > 05:32:24.210315 IP 195.95.178.204.80 > 94.243.104.156.32924: 
>> > > Flags [.], seq 387:1791, ack 658, win 237, options [nop,nop,TS 
>> > > val
>> > > 05:32:24.210332 IP 195.95.178.204.80 > 94.243.104.156.32924: 
>> > > Flags [.], seq 1791:3195, ack 658, win 237, options [nop,nop,TS 
>> > > val
>> > > 05:32:24.210347 IP 195.95.178.204.80 > 94.243.104.156.32924: 
>> > > Flags [.], seq 3195:4599, ack 658, win 237, options [nop,nop,TS 
>> > > val
>> > > 05:32:24.210350 IP 94.243.104.156.32924 > 195.95.178.204.80: 
>> > > Flags [.], ack 387, win 347, options [nop,nop,TS val 1112359 ecr 
>> > > 2808808677], length 0
>> > > 05:32:24.210361 IP 195.95.178.204.80 > 94.243.104.156.32924: 
>> > > Flags [.], seq 4599:6003, ack 658, win 237, options [nop,nop,TS 
>> > > val
>> > > 05:32:24.210374 IP 195.95.178.204.80 > 94.243.104.156.32924: 
>> > > Flags [.], seq 6003:7407, ack 658, win 237, options [nop,nop,TS 
>> > > val
>> > > 05:32:24.210389 IP 195.95.178.204.80 > 94.243.104.156.32924: 
>> > > Flags [.], seq 7407:8811, ack 658, win 237, options [nop,nop,TS 
>> > > val
>> > > 05:32:24.210402 IP 195.95.178.204.80 > 94.243.104.156.32924: 
>> > > Flags [.], seq 8811:10215, ack 658, win 237, options [nop,nop,TS 
>> > > val
>> > > 05:32:24.210416 IP 195.95.178.204.80 > 94.243.104.156.32924: 
>> > > Flags [.], seq 10215:11619, ack 658, win 237, options [nop,nop,TS 
>> > > val
>> > > 05:32:24.210418 IP 94.243.104.156.32924 > 195.95.178.204.80: 
>> > > Flags [.], ack 1791, win 358, options [nop,nop,TS val 1112359 ecr 
>> > > 2808808677], length 0
>> > > 05:32:24.210431 IP 195.95.178.204.80 > 94.243.104.156.32924: 
>> > > Flags [.], seq 11619:13023, ack 658, win 237, options [nop,nop,TS 
>> > > val
>> > > 05:32:24.210455 IP 94.243.104.156.32924 > 195.95.178.204.80: 
>> > > Flags [.], ack 3195, win 369, options [nop,nop,TS val 1112359 ecr 
>> > > 2808808677], length 0
>> > > 05:32:24.210685 IP 94.243.104.156.32924 > 195.95.178.204.80: 
>> > > Flags [.], ack 4599, win 380, options [nop,nop,TS val 1112359 ecr 
>> > > 2808808677], length 0
>> > > 05:32:24.210731 IP 94.243.104.156.32924 > 195.95.178.204.80: 
>> > > Flags [.], ack 6003, win 391, options [nop,nop,TS val 1112359 ecr 
>> > > 2808808677], length 0
>> > > 05:32:24.210937 IP 94.243.104.156.32924 > 195.95.178.204.80: 
>> > > Flags [.], ack 7407, win 402, options [nop,nop,TS val 1112359 ecr 
>> > > 2808808677], length 0
>> > > 05:32:24.211078 IP 94.243.104.156.32924 > 195.95.178.204.80: 
>> > > Flags [.], ack 8811, win 413, options [nop,nop,TS val 1112359 ecr 
>> > > 2808808677], length 0
>> > > 05:32:24.211228 IP 94.243.104.156.32924 > 195.95.178.204.80: 
>> > > Flags [.], ack 10215, win 424, options [nop,nop,TS val 1112359 
>> > > ecr 2808808677], length 0
>> > > 05:32:24.211377 IP 94.243.104.156.32924 > 195.95.178.204.80: 
>> > > Flags [.], ack 11619, win 435, options [nop,nop,TS val 1112359 
>> > > ecr 2808808677], length 0
>> > > 05:32:24.211544 IP 94.243.104.156.32924 > 195.95.178.204.80: 
>> > > Flags [.], ack 13023, win 446, options [nop,nop,TS val 1112359 
>> > > ecr 2808808677], length 0
>> > > 05:32:24.237999 IP 195.95.178.204.80 > 94.243.104.156.32924: 
>> > > Flags [.], seq 13023:14427, ack 658, win 237, options [nop,nop,TS 
>> > > val
>> > > 05:32:24.238053 IP 195.95.178.204.80 > 94.243.104.156.32924: 
>> > > Flags [.], seq 14427:15831, ack 658, win 237, options [nop,nop,TS 
>> > > val
>> > > 05:32:24.238090 IP 94.243.104.156.32924 > 195.95.178.204.80: 
>> > > Flags [.], ack 14427, win 457, options [nop,nop,TS val 1112362 
>> > > ecr 2808808709], length 0
>> > > 05:32:24.238164 IP 94.243.104.156.32924 > 195.95.178.204.80: 
>> > > Flags [.], ack 15831, win 468, options [nop,nop,TS val 1112362 
>> > > ecr 2808808709], length 0
>> > > 05:32:24.244024 IP 195.95.178.204.80 > 94.243.104.156.32924: 
>> > > Flags [.], seq 15831:17235, ack 658, win 237, options [nop,nop,TS 
>> > > val
>> > > 05:32:24.244063 IP 195.95.178.204.80 > 94.243.104.156.32924: 
>> > > Flags [.], seq 17235:18639, ack 658, win 237, options [nop,nop,TS 
>> > > val

---

From: Ingemar Johansson S <[email-masked]>
Date: Thu, 1 Oct 2015 17:02:08 +0000

Hi

An increased threshold would probably solve the problem and is possibly the quick fix. 
I will anyway need to rerun parts of my simulations because of my own 32ms mistake, it is quite straightforward to add a few alternative threshold values in the same work.

Wonder though if one can do it smarter?.
For instance, assuming that the timestamp option is enabled, it should there be possible to infer if ACKs belong to the same HyStart round, based on TSval. 
It of course adds some extra lines of code to the HyStart algorithm but then one can perhaps keep the current thresholds ?.

/Ingemar

> Sent: den 1 oktober 2015 17:17
> To: Eric Dumazet; Ingemar Johansson S
> Cc: Eric Dumazet; Neal Cardwell; [email-masked]; Piers O'Hanlon; Sangtae Ha;
> [email-masked]
> Sure 10 ms will be better as more of the LTE technological jumps will be
> covered and, at least, will help get rid of exit at the minimum exit cwnd (29,
> which at 20 ms RTT is ~16Mbps) and will shift to 49 (~28 Mbps) or to 89 (~ 50
> Mbps).
> Cc: Veaceslav ROMAN; Eric Dumazet; Neal Cardwell; [email-masked]; Piers
> O'Hanlon; Sangtae Ha; [email-masked]
> Yes, this 4ms value problem is even more visible with TCP pacing (as
> implemented with linux sch_fq packet scheduler) because of apparent
> delays caused by sojourn time in packet scheduler itself.
> TCP does not offset this sojourn time (as RTT is also used for RTO
> determination, better have an inflated view)
> 
> On Thu, Oct 1, 2015 at 4:57 AM, Ingemar Johansson S
> <[email-masked]> wrote:
> > Hmm
> > You are right !.
> >> > From: Ingemar Johansson S
> [mailto:[email-masked]]
> >> > > r3_8 as a min(ack_rtt_31 ... ack_rtt_38) and ), r4_8 as a min(ack_rtt_71
> ...
> >> > ack_rtt_78).
> >> > >

---

From: Veaceslav ROMAN <[email-masked]>
Date: Fri, 2 Oct 2015 11:53:28 +0000

Thank you. I'll be grateful if you share your further results.
As for timestamp, possibly they can be used for LTE, but because of they granularity of 1, 4 or 10 ms I am not sure they will be enough for sub millisecond neyworks.

BTW, the server used in tests I described below sends timestamps every 4 ms and the client sends every 10 ms.   

Veaceslav Roman

-----Original Message-----
From: Ingemar Johansson S [mailto:[email-masked]] 
Sent: Thursday, 01 October 2015 20:02
To: Veaceslav ROMAN; Eric Dumazet
Cc: Eric Dumazet; Neal Cardwell; [email-masked]; Piers O'Hanlon; Sangtae Ha; [email-masked]; Ingemar Johansson S
Subject: RE: [tcpm] [e2e] TCP HyStart patch deployment

Hi

/Ingemar

> Cc: Eric Dumazet; Neal Cardwell; [email-masked]; Piers O'Hanlon; 
> Sangtae Ha; [email-masked]
> Sure 10 ms will be better as more of the LTE technological jumps will 
> be covered and, at least, will help get rid of exit at the minimum 
> exit cwnd (29, which at 20 ms RTT is ~16Mbps) and will shift to 49 
> (~28 Mbps) or to 89 (~ 50 Mbps).
> >> less than 32<<3, then the delay_min>>3 will be less than 32 and 
> >> will compare with
> >> Yes, the variable delay_min is in Q3 for reasons of precision I 
> >> guess But the call to HYSTART_DELAY_THRESH is with delay_min 
> >> shifted down to Q0
> >>
> >> > Regarding pacing I am not sure will help the performance: there 
> >> > are gaps in ACK trains of the order of milliseconds and this is a 
> >> > lost time and there is no way to compensate it. If introduce ACK 
> >> > pacing or data packets pacing this would add to the overall lost time.
> >> > unexpected packets drops when large train sent at a high speed as 
> >> > a reply to highly compressed ACK train, but, on another hands.
> >> > time but in predefined for him intervals. In my network this 
> >> > period is 5 ms, therefore the RTT of the first ACK in the train 
> >> > will have variations of 2.5 ms. But standards admit this period 
> >> > to be also 10 ms ,20 ms, 40 ms, 80 ms and I can not say what is 
> >> > the practice of others. I may only guess that most will try to 
> >> > use
> >> > Anyway, to get a realistic modeling of LTE interaction with TCP 
> >> > the detailed MAC scheduler model is needed. Unfortunately, most 
> >> > of the literature focus on scheduling of the Physical Resource 
> >> > Blocks, Adaptive Modulation and Coding, MIMO, i.e., focusing on 
> >> > the distribution of resources between users, which is good and 
> >> > needed, but do not address very much the timing interaction with 
> >> > TCP.  At least, I never met in LTE scheduling literature a term "TCP self-clocking".
> >> >
> >> > The 8 ms  HYSTART_DELAY_MIN could only be a simple solution for 
> >> > the problem of counting of the round' curr_rtt from the second 
> >> > ACK of the train (as the second ACK RTT will be, typically, due 
> >> > to scheduling,
> >> > inside out and upside down but sofar it has been very difficult 
> >> > to make HyStart exit prematurely but then of course I have run 
> >> > with HYSTART_DELAY_MIN = 32ms.
> >> >
> >> > easily occur in LTE, even in simulations. What I have seen is 
> >> > that packet pacing is a very efficient remedy, for instance the 
> >> > packet implemented in QUIC seems to solve ACK compression issues very well.
> >> >
> >> > up, I could understand that they are split up in two parts 
> >> > (initial grant + additional after received BSR).
> >> >
> >> > /Ingemar
> >> >
> >> > > Cc: [email-masked]; Piers O'Hanlon; Sangtae Ha; Ingemar 
> >> > > Johansson S; Eric Dumazet; [email-masked]
> >> > >            net.ipv4.tcp_no_metrics_save = 1 to avoid tcp 
> >> > > metrics caching interference
> >> > >
> >> > >   Tests types:
> >> > > client side as shown by curl from the first received data 
> >> > > packet till the end of session (this exclude DNS, 3 way 
> >> > > handshake, http GET and its
> >> ACK).
> >> > > conditions of low available bandwidth, but at high available 
> >> > > the decrease of performance can reach 60-70 %.
> >> > > I do share with Eric and Neal some files described below and 
> >> > > also can provide links to anyone interested.
> >> > >
> >> > > pcap (able to collect only server side traces, but they are 
> >> > > self
> sufficient).
> >> > > For the 10M downloads there are few missing traces as there 
> >> > > trace files were recycled. In traces are also ssh sessions, 
> >> > > please, ignore them, they were used to put hystart ON and OFF 
> >> > > on server and extract nstat results before and after each download.
> >> > >
> >> > > In the curl_sum tab the stat is cumulated in chronological 
> >> > > order of
> tests.
> >> > > also include graphs of Transfer time and a graph which shows 
> >> > > the distribution in % of different Hystart exit window DelayCWND.
> >> > >
> >> > > Fields:
> >> > > comments  - test case
> >> > > Then there are calculated fields: minRTT as a minimum of all 
> >> > > 100 ACK,
> >> r1_5,
> >> > > and r1 min, r2 min, r3 min, r4 min represent minRTT of the 
> >> > > whole
> round.
> >> > > train the first packet has the lowest RTT which, most probably 
> >> > > is valid in all type of networks, then the exit from hystart in 
> >> > > this implementation may happen at cwnd 29, 49, 89, 169, etc. 
> >> > > (given the IW of
> >> > 10).
> >> > > But there are also intermediate values, e.g. 69. Looking 
> >> > > through traces one may see that this is due to "compressed ACK" 
> >> > > which receives a tightly packed train of many ACK so that in 
> >> > > traces there are not data packets sent in turn by server. Looks 
> >> > > to me, because hysrart_reset sets the end_seq to snd_next and 
> >> > > the server did not managed to send yet packets in reply to this 
> >> > > high speed ACK train the snd_next points to much lower value 
> >> > > than would normally happen and this actually destroys the 
> >> > > correct identification of the borders of the round, the round 
> >> > > fails somewhere in the middle of the train not at its border, 
> >> > > this makes very likely increased delay due to queueing and 
> >> > > earlier exit from slow_start and intermediate values of exit 
> >> > > cwnd where
> >> > they are not expected.
> >> > >
> >> > > And last but not least: there are gaps (4-7 ms) in ACK trains. 
> >> > > On the perfectly sent IW of 10 data packets train the typical 
> >> > > for LTE will be to receive 1 or 2 ACK, then a gap of 4-6 ms 
> >> > > then another couple of ACK, then another gap, then 5 ACK 
> >> > > compressed together in a fraction of microsecond. Of cause, the 
> >> > > next train from the server will contain the same gaps. Very 
> >> > > quickly, e.g., the Head of Line
> >> > > (actually the radio base station, much earlier than BDP 
> >> > > reached), but this will lead to increase of the train RTT and 
> >> > > too yearly exit from slow start. Why too early exit ? Because, 
> >> > > should server continue increasing the speed this would reduce 
> >> > > quicker the gaps both in downlink and uplink, the preice for 
> >> > > this being the increase of the RTT (I guess double) against 
> >> > > what one would normally see
> >> > > LTE speeds this early exit from slow start will become even 
> >> > > more of the
> >> problem.
> >> > > some operators use, would be to insert a kind of TCP 
> >> > > accelerator between the Internet and the mobile network which 
> >> > > will intercept all TCP and would send it to mobile without 
> >> > > Hystart but this would kill any end-to-end principle and this 
> >> > > will not be the Internet
> anymore.
> >> > >
> >> > >
> >> > > If you have time and never look closely to LTE technology I do 
> >> > > a quick summary bellow.
> >> > >
> >> > >
> >> > >   First, unlike wire based transport technologies, LTE (but 
> >> > > also
> >> > UMTS
> >> > > and future 5G) sends data, both uplink and downlink in 
> >> > > predefined timeslots called Transmission Time Interval (TTI). 
> >> > > In UMTS this is 2ms, in LTE is
> >> > > require exact synchronization between the mobile device and 
> >> > > Radio Base Station (called eNodeB in LTE). To cope with varying 
> >> > > radio conditions the sending data bandwidth is modified in such 
> >> > > a way that to keep a constant probability of data loss of 10%. 
> >> > > Bad radio - less bandwidth, good
> >> > radio - more bandwidth.
> >> > > But TTI remain unchanged of exactly 1 ms. That means that, at 
> >> > > the TCP/IP layer the precision of the timing information from 
> >> > > device to the network is 1 ms. In the case of a bandwidth of 1 
> >> > > Mbps there will be 83 IP packets per second or one packet every 12 ms.
> >> > > 12 TTI) would be more or less sufficient to keep at a good 
> >> > > level the TCP self-clocking mechanism. But, when talking about 
> >> > > 100 Mbps (or 300
> >> > > Mbps) that means one packet (and one ACK) every 0.12 ms and 
> >> > > there is no way to provide it with a TTI of 1 ms. This is the 
> >> > > reason for ACK compression in LTE and the future 5G which 
> >> > > promises more than 1Gbps with its 0.5 ms TTI will
> >> > > (eNodeB is a kind of IP wireless router). But for mobile 
> >> > > device, in order to send the data in uplink, after a pause, it 
> >> > > has first to ask the network to schedule it. And mobile has for 
> >> > > this request (after a pause of
> >> > > In good radio conditions the eNodeB will fit all 10 packets in 
> >> > > 1 TTI of 1 ms and send them all to the device. Device will 
> >> > > unpack them and generate 10 ACK and then tell to the eNodeB 
> >> > > "Hey, I have some data in the buffer to send". The eNodeB has 
> >> > > not knowledge how much data the mobile has to send, therefor 
> >> > > will allocate some minimum
> >> > > 4 ms are allowance for mobile device to prepare data for 
> >> > > sending which is a very complicated process, these 4 ms is a 
> >> > > must delay between the grant
> >> > and the transmission.
> >> > > report: "I do have still 480 bytes to send". Now network has 
> >> > > more information and allocate as
> >> > > requested: "Well, I give you a grant to send up to 200 Bytes 
> >> > > and you must send them exactly after 4 ms you get this notification".
> >> > > signaling) and the device will send only 6 out of 8 remaining 
> >> > > ACK together with signaling and another Buffer Status Report: 
> >> > > "Hey, I do have another 120 bytes to send". And again "Well, I 
> >> > > give you a grant to send up to 120 Bytes and you must send them 
> >> > > exactly after 4 ms you get this notification". As a result the 
> >> > > server will receive 2 ACK, then after a gap of 6 ms another 6 
> >> > > ACK, then, after another 6 ms, the remaining 2 ACK. With the 
> >> > > scheduler in the eNodeB these gaps are inevitable. And one more 
> >> > > think: mobile pack together those 6 ACK in 1 TTI of 1 ms and 
> >> > > send them, then the eNodeB unpack them and there is no way to 
> >> > > recover the spacing
> of 0.12 ms between them.
> >> > > compression. After the last 2 ACK were sent there is nothing 
> >> > > else to send, so there is no Buffer Status Report, therefore, 
> >> > > after packets of the next train arrive and ACK are generated 
> >> > > everything starts again with one bit of information "Hey, I 
> >> > > have
> >> > > exactly, lack of any timing information which would be useful 
> >> > > for the TCP self-
> >> > clocking.
> >> > >
> >> > > Cc: [email-masked]; Piers O'Hanlon; Sangtae Ha; Ingemar 
> >> > > Johansson S; Eric Dumazet; [email-masked]
> >> > > called transmission time interval (TTI) of 1 ms, and what we 
> >> > > see here is that all 10 packets of initial window fitted in 1 
> >> > > ms, and, when decoded, were presented to the TCP/IP layer (and 
> >> > > pcap) all
> at once.
> >> > > should be 20) packets reaches a duration of 14 ms. AFAIK LTE 
> >> > > may introduce gaps in ACK train due to uplink scheduling mechanism.
> >> > > > I agree that in your LTE traces it looks like CUBIC Hystart 
> >> > > > is exiting slow start too early.
> >> > > >
> >> > > > SInce you seem to have a nice LTE testbed, would you be able 
> >> > > > to do some experiments to find a set of parameters for 
> >> > > > Hystart that work better for your LTE environment? For 
> >> > > > example, you might try the two variations I suggested earlier in the thread:
> >> > > >
> >> > > This problem would be solved if GRO was used at receiver, as 
> >> > > less ACK would be sent.
> >> > >
> >> > > Maybe we should add a logic in hystart_update() to take one ACK 
> >> > > per
> ms.
> >> > >
> >> > > Flags [S], seq 4294423215, win 65535, options [mss 
> >> > > 1460,sackOK,TS val
> >> > > Flags [.], ack 387, win 347, options [nop,nop,TS val 1112359 
> >> > > Flags [.], seq 8811:10215, ack 658, win 237, options 
> >> > > [nop,nop,TS val
> >> > > Flags [.], seq 10215:11619, ack 658, win 237, options 
> >> > > [nop,nop,TS val
> >> > > Flags [.], ack 1791, win 358, options [nop,nop,TS val 1112359 
> >> > > Flags [.], seq 11619:13023, ack 658, win 237, options 
> >> > > [nop,nop,TS val
> >> > > Flags [.], ack 3195, win 369, options [nop,nop,TS val 1112359 
> >> > > Flags [.], ack 4599, win 380, options [nop,nop,TS val 1112359 
> >> > > Flags [.], ack 6003, win 391, options [nop,nop,TS val 1112359 
> >> > > Flags [.], ack 7407, win 402, options [nop,nop,TS val 1112359 
> >> > > Flags [.], ack 8811, win 413, options [nop,nop,TS val 1112359 
> >> > > Flags [.], seq 13023:14427, ack 658, win 237, options 
> >> > > [nop,nop,TS val
> >> > > Flags [.], seq 14427:15831, ack 658, win 237, options 
> >> > > [nop,nop,TS val
> >> > > Flags [.], seq 15831:17235, ack 658, win 237, options 
> >> > > [nop,nop,TS val
> >> > > Flags [.], seq 17235:18639, ack 658, win 237, options 
> >> > > [nop,nop,TS val

---

From: Veaceslav ROMAN <[email-masked]>
Date: Mon, 5 Oct 2015 22:07:02 +0000

We've managed to recompile the kernel 4.1 with tcp_cubic HYSTART_DELAY_MIN  (10U<<3) , 10ms.
Well, we had time only for one test cycle, but results are very promising:

Download 10MB, High Bandwidths, HYSTART_DELAY_MIN  10 ms:
           Hystart average download time: 1.16 s
           NoHystart average download time: 0.93  s

           Avr. DelayCWND: 159
            Min DelayCWND: 49 
           Max DelayCWND 754 
           Of all 50 tests with Hystart, HystartDelay detected in all 50 tests. 

DelayCWND: 49, 69, 73, 75, 77, 81, 83, 84, 85, 87, 88, 89, 133, 137, 139, 145, 150, 153, 160, 162, 165, 301, 307, 321, 454, 640, 707, 754
Cases:               4,    1,  1,    1,   2,  2,   2,   1,   7,   3,   2,   7,     1,     1,     2,      1,     1,     1,     1,     1,     1,     1,     1,    1,      1,     1,     1,     1

This shows an improvement of download time 20% in comparison with HYSTART_DELAY_MIN  4 ms and reduced the gap with NoHystart conditions from -59%  (HDM 4ms) to -25% (HDM 10 ms). 

Compare with previous round with HYSTART_DELAY_MIN  of 4 ms 

 Download 10MB, High Bandwidths, HYSTART_DELAY_MIN  4 ms:
           Hystart average download time: 1.42 s
           NoHystart average download time: 0.89  s

           Avr. DelayCWND: 60
           Min DelayCWND: 29
           Max DelayCWND 89 
           Of all 50 tests with Hystart, HystartDelay detected in all 50 tests. 
 
DelayCWND: 29, 45, 49, 63, 69, 81, 83, 85, 86, 87, 88, 89
Cases:              13,  1, 12,   1,   3,   2,   2,   1,   2,   6,   1,   6

I'll be in the trip till the end of the week, will be able to come with more details next week. 

Best regards

Veaceslav Roman

________________________________________
From: Eric Dumazet [[email-masked]]
Sent: Thursday, October 01, 2015 3:44 PM
To: Ingemar Johansson S
Cc: Veaceslav ROMAN; Eric Dumazet; Neal Cardwell; [email-masked]; Piers O'Hanlon; Sangtae Ha; [email-masked]
Subject: Re: [tcpm] [e2e] TCP HyStart patch deployment

Yes, this 4ms value problem is even more visible with TCP pacing
(as implemented with linux sch_fq packet scheduler)
because of apparent delays caused by sojourn time in packet scheduler itself.
TCP does not offset this sojourn time (as RTT is also used for RTO
determination, better have an inflated view)

---

From: Neal Cardwell <[email-masked]>
Date: Mon, 5 Oct 2015 21:05:56 -0400

...
>            NoHystart average download time: 0.89  s

Thanks! That is very useful and interesting data. Would you be able to
try a few other values, like 15ms and 20ms?

neal

---

From: Veaceslav ROMAN <[email-masked]>
Date: Wed, 7 Oct 2015 10:58:04 +0000

Yes, we'll test it too, but next week.
It will be very much helpful if someone can rewrite tcp_cubic.c to make  HYSTART_DELAY_MIN externally settable, otherwise, according to our IT it takes almost a whole working day to recompile the kernel.

Best regards

Veaceslav Roman

________________________________________
From: Neal Cardwell [[email-masked]]
Sent: Tuesday, October 06, 2015 4:05 AM
To: Veaceslav ROMAN
Cc: Eric Dumazet; Ingemar Johansson S; Eric Dumazet; [email-masked]; Piers O'Hanlon; Sangtae Ha; [email-masked]
Subject: Re: [tcpm] [e2e] TCP HyStart patch deployment

...

neal

---

From: Veaceslav ROMAN <[email-masked]>
Date: Sat, 17 Oct 2015 20:48:26 +0000

Hi,
Recompiled kernel to HYSTART_DELAY_MIN  20 ms: 

  Download 10MB, High Bandwidths, HYSTART_DELAY_MIN  20 ms:
           Hystart average download time: 1.18 s
           NoHystart average download time: 1.00  s

No significant improvement against the case with HYSTART_DELAY_MIN  10 ms.

Sill Hystart early exit is visible. 

Other tests with HYSTART_DELAY_MIN  20 ms and files of 3MB.

  Download 3MB, High Bandwidths, HYSTART_DELAY_MIN  20 ms:
           Hystart average download time: 0.47 s
           NoHystart average download time: 0.39  s

compare with 4 ms

  Download 3MB, High Bandwidths, HYSTART_DELAY_MIN  4 ms:
           Hystart average download time: 0.65 s
           NoHystart average download time: 0.37  s

Few more comment: High Bandwidth is typically above 30-40 Mbps.
For Low Bandwidth, when radio do not permit above 40 Mbps Hystart early exit does not have a significant impact. I do explain it with the fact that the minimum exit CWND for Hysstart is 29 which, for RTT of ~15ms (minimum possible in LTE and visible in majority of traces) leads to ~25 Mbps, so Hystart exit, at least, at 25 Mbps and then it takes not too long to cubic to reach 30-40 Mbps.  

Veaceslav Roman

-----Original Message-----
From: Neal Cardwell [mailto:[email-masked]] 
Sent: Tuesday, 06 October 2015 04:06
To: Veaceslav ROMAN
Cc: Eric Dumazet; Ingemar Johansson S; Eric Dumazet; [email-masked]; Piers O'Hanlon; Sangtae Ha; [email-masked]
Subject: Re: [tcpm] [e2e] TCP HyStart patch deployment

...
>            NoHystart average download time: 0.89  s

Thanks! That is very useful and interesting data. Would you be able to try a few other values, like 15ms and 20ms?

neal

---

From: Ingemar Johansson S <[email-masked]>
Date: Wed, 21 Oct 2015 12:58:56 +0000

Hi Bob
Yes, path chirp and RAPID is quite interesting, one thing that I wonder about is how this will work in cases where access technology adds jitter.  
One such example is LTE, whose properties I tried to outline in 
https://tools.ietf.org/html/draft-johansson-cc-for-4g-5g-00 

Promised to come back with simulation results on different HyStart threshold values, 
Time is however a sparse commodity it seems, and besides that I feel I need to back up possible simulation results with some real data.

/Ingemar

> From: Bob Briscoe [mailto:[email-masked]]
> Sent: den 15 oktober 2015 17:23
> To: Ingemar Johansson S; Veaceslav ROMAN; Eric Dumazet
> Cc: [email-masked]; Piers O'Hanlon; Sangtae Ha; end2end-
> [email-masked]
> Ingemar,
> On 01/10/15 18:02, Ingemar Johansson S wrote:
> > Hi
> >
> > I will anyway need to rerun parts of my simulations because of my own
> 32ms mistake, it is quite straightforward to add a few alternative threshold
> values in the same work.
> >
> > For instance, assuming that the timestamp option is enabled, it should
> there be possible to infer if ACKs belong to the same HyStart round, based
> on TSval.
> > It of course adds some extra lines of code to the HyStart algorithm but then
> one can perhaps keep the current thresholds ?.
> Re: doing it smarter, Mirja & I wrote up the following about how to use high
> precision timestamps in Linux to rapidly get info on available capacity with
> minimum intrusiveness using packet chirps (based on the ideas in pathchirp
> and TCP RAPID):
> <http://www.bobbriscoe.net/pubs.html#chirp_impl>
> The paper doesn't focus particularly on slow-start, but that's usually the most
> useful case.

> 
> BTW, we were (mis)using TCP timestamps to infer one-way delay, so we had
> to arrange for both ends to be writing high precision timestamps into the TS
> option. But if you are happy with round-trip delay, host A can put a high
> precision timestamp in the TSval field, and the receiver will dumbly reflect it
> in the TSecr field without trying to understand its precision. See S.3.1.
> 
> 
> Bob
> >
> > /Ingemar
> >
> >> Cc: Veaceslav ROMAN; Eric Dumazet; Neal Cardwell; [email-masked];
> >> Piers O'Hanlon; Sangtae Ha; [email-masked]
> >>> Well... That of course should explain why HyStart performs badly for
> LTE...
> >> Atleast it is part of the reason.
> >>> /Ingemar
> >>>
> >>>
> >>>
> >>>>> will have variations of 2.5 ms. But standards admit this period to
> >>>>> be also 10 ms ,20 ms, 40 ms, 80 ms and I can not say what is the
> >>>>> practice of others. I may only guess that most will try to use
> >>>>> the detailed MAC scheduler model is needed. Unfortunately, most of
> >>>>> the literature focus on scheduling of the Physical Resource
> >>>>> the problem of counting of the round' curr_rtt from the second ACK
> >>>>> of the train (as the second ACK RTT will be, typically, due to
> >>>>> scheduling,
> >>>>>> In the curl_sum tab the stat is cumulated in chronological order
> >>>>>> of
> >> tests.
> >>>>>> would kill any end-to-end principle and this will not be the
> >>>>>> Internet
> >> anymore.
> >>>>>>
> there
> >>>>>> and, when decoded, were presented to the TCP/IP layer (and pcap)
> >>>>>> all
> >> at once.
> >>>>>>> do some experiments to find a set of parameters for Hystart that
> >>>>>>> work better for your LTE environment? For example, you might try
> >>>>>>> the two variations I suggested earlier in the thread:
> >>>>>>>
> > _______________________________________________
> > tcpm mailing list
> > [email-masked]
> > https://www.ietf.org/mailman/listinfo/tcpm
> >
> 
> --
> __________________________________________________________
> ______
> Bob Briscoe                               http://bobbriscoe.net/