HyStart++: Modified Slow Start for TCPConfluent899 West Evelyn AveMountain ViewCA94041United States of Americapravb.ietf@gmail.comMicrosoftOne Microsoft WayRedmondWA98052United States of America+1 425 703 0447huanyi@microsoft.comMicrosoftOne Microsoft WayRedmondWA98052United States of America+1 425 538 8598maolson@microsoft.com
tsv
tcpmTCPcongestion control This document describes HyStart++, a simple modification to
the slow start phase of congestion control algorithms.
Slow start can overshoot the ideal send rate
in many cases, causing high packet loss and poor performance.
HyStart++ uses increase in round-trip delay as a heuristic to
find an exit point before possible overshoot.
It also adds a mitigation to prevent jitter from causing
premature slow start exit.
Status of This Memo
This is an Internet Standards Track document.
This document is a product of the Internet Engineering Task Force
(IETF). It represents the consensus of the IETF community. It has
received public review and has been approved for publication by
the Internet Engineering Steering Group (IESG). Further
information on Internet Standards is available in Section 2 of
RFC 7841.
Information about the current status of this document, any
errata, and how to provide feedback on it may be obtained at
.
Copyright Notice
Copyright (c) 2023 IETF Trust and the persons identified as the
document authors. All rights reserved.
This document is subject to BCP 78 and the IETF Trust's Legal
Provisions Relating to IETF Documents
() in effect on the date of
publication of this document. Please review these documents
carefully, as they describe your rights and restrictions with
respect to this document. Code Components extracted from this
document must include Revised BSD License text as described in
Section 4.e of the Trust Legal Provisions and are provided without
warranty as described in the Revised BSD License.
Table of Contents
. Introduction
. Terminology
. Definitions
. HyStart++ Algorithm
. Summary
. Algorithm Details
. Tuning Constants and Other Considerations
. Deployments and Performance Evaluations
. Security Considerations
. IANA Considerations
. References
. Normative References
. Informative References
Acknowledgments
Authors' Addresses
Introduction describes the slow start
congestion control algorithm for TCP. The slow start
algorithm is used when the congestion window (cwnd)
is less than the slow start threshold (ssthresh).
During slow start, in the absence of packet loss signals,
TCP increases the cwnd exponentially to probe the network capacity.
This fast growth can overshoot the ideal sending rate
and cause significant packet loss that cannot always
be recovered efficiently.
HyStart++ builds upon Hybrid Start (HyStart), originally described in
. HyStart++ uses increase in
round-trip delay as a signal to exit
slow start before potential packet loss occurs as a result
of overshoot. This is one of two algorithms specified in
for finding a safe exit point for
slow start. After the slow start exit, a new
Conservative Slow Start (CSS) phase is used to determine
whether the slow start exit was premature and to resume
slow start. This mitigation improves performance in the
presence of jitter.
HyStart++ reduces packet loss and retransmissions, and
improves goodput in lab measurements and real-world
deployments.
While this document describes HyStart++ for TCP, it can
also be used for other transport protocols that use slow start,
such as QUIC
or the Stream Control Transmission Protocol (SCTP) .
TerminologyThe key words "MUST", "MUST NOT",
"REQUIRED", "SHALL",
"SHALL NOT", "SHOULD",
"SHOULD NOT",
"RECOMMENDED", "NOT RECOMMENDED",
"MAY", and "OPTIONAL" in this document
are to be interpreted as described in BCP 14
when, and only
when, they appear in all capitals, as shown here.Definitions To aid the reader, we repeat some definitions from
:
SENDER MAXIMUM SEGMENT SIZE (SMSS):
The size of the
largest segment that the sender can transmit. This value can be
based on the maximum transmission unit of the network, the Path
MTU Discovery algorithm , RMSS (see next item),
or other factors. The size does not include the TCP/IP headers
and options.
RECEIVER MAXIMUM SEGMENT SIZE (RMSS):
The
size of the largest segment that the receiver is willing to accept.
This is the value specified in the MSS option sent by the
receiver during connection startup. Or, if the MSS option
is not used, it is 536 bytes .
The size does not include the TCP/IP headers and
options.
RECEIVER WINDOW (rwnd):
The most recently advertised
receiver window.
CONGESTION WINDOW (cwnd):
A TCP state variable that
limits the amount of data a TCP can send.
At any given time, a TCP MUST NOT send
data with a sequence number higher than the sum of the highest
acknowledged sequence number and the minimum of the cwnd and rwnd.
HyStart++ AlgorithmSummary specifies two algorithms
(a "Delay Increase" algorithm and an "Inter-Packet Arrival"
algorithm) to be run in parallel to detect that the sending
rate has reached capacity. In practice, the Inter-Packet
Arrival algorithm does not perform well and is not able
to detect congestion early, primarily due to ACK compression.
The idea of the Delay Increase algorithm is to look for
spikes in RTT (round-trip time), which suggest that the
bottleneck buffer is filling up.
In HyStart++, a TCP sender uses standard slow start
and then uses the Delay Increase algorithm to trigger an
exit from slow start. But instead of going straight from
slow start to congestion avoidance, the sender spends a
number of RTTs in a Conservative Slow Start (CSS) phase
to determine whether the exit from slow start was premature.
During CSS, the congestion window is grown exponentially in a
fashion similar to regular slow start, but with a smaller exponential base,
resulting in less aggressive growth.
If the RTT reduces during CSS, it's concluded that the RTT
spike was not related to congestion caused by the connection
sending at a rate greater than the ideal send rate, and the
connection resumes slow start. If the RTT inflation
persists throughout CSS, the connection enters congestion
avoidance.
Algorithm Details The following pseudocode uses a limit, L, to control the
aggressiveness of the cwnd increase during both standard slow
start and CSS. While an arriving ACK may newly acknowledge an
arbitrary number of bytes, the HyStart++ algorithm limits the
number of those bytes applied to increase the cwnd to L*SMSS bytes. lastRoundMinRTT and currentRoundMinRTT are initialized
to infinity at the initialization time. currRTT is the RTT
sampled from the latest incoming ACK and initialized to
infinity.
lastRoundMinRTT = infinity
currentRoundMinRTT = infinity
currRTT = infinity
HyStart++ measures rounds using sequence numbers, as
follows:
Define windowEnd as a sequence number initialized to SND.NXT.
When windowEnd is ACKed, the current round ends and windowEnd
is set to SND.NXT.
At the start of each round during standard slow start
and CSS, initialize the variables
used to compute the last round's and current round's minimum RTT:
lastRoundMinRTT = currentRoundMinRTT
currentRoundMinRTT = infinity
rttSampleCount = 0
For each arriving ACK in slow start, where N is the
number of previously unacknowledged bytes acknowledged
in the arriving ACK: Update the cwnd:
cwnd = cwnd + min(N, L * SMSS)
Keep track of the minimum observed RTT:
currentRoundMinRTT = min(currentRoundMinRTT, currRTT)
rttSampleCount += 1
For rounds where at least N_RTT_SAMPLE RTT samples have been
obtained and currentRoundMinRTT and lastRoundMinRTT are valid,
check to see if delay increase triggers slow start exit:
if ((rttSampleCount >= N_RTT_SAMPLE) AND
(currentRoundMinRTT != infinity) AND
(lastRoundMinRTT != infinity))
RttThresh = max(MIN_RTT_THRESH,
min(lastRoundMinRTT / MIN_RTT_DIVISOR, MAX_RTT_THRESH))
if (currentRoundMinRTT >= (lastRoundMinRTT + RttThresh))
cssBaselineMinRtt = currentRoundMinRTT
exit slow start and enter CSS
For each arriving ACK in CSS, where N is the number
of previously unacknowledged bytes acknowledged in
the arriving ACK: Update the cwnd:
cwnd = cwnd + (min(N, L * SMSS) / CSS_GROWTH_DIVISOR)
Keep track of the minimum observed RTT:
currentRoundMinRTT = min(currentRoundMinRTT, currRTT)
rttSampleCount += 1
For CSS rounds where at least N_RTT_SAMPLE RTT
samples have been obtained, check to see if the current round's
minRTT drops below baseline (cssBaselineMinRtt) indicating that
slow start exit was spurious:
if (currentRoundMinRTT < cssBaselineMinRtt)
cssBaselineMinRtt = infinity
resume slow start including HyStart++
CSS lasts at most CSS_ROUNDS rounds. If the transition
into CSS happens in the middle of a round, that partial
round counts towards the limit. If CSS_ROUNDS rounds are complete,
enter congestion avoidance by setting the ssthresh to the current cwnd.
ssthresh = cwnd
If loss or Explicit Congestion Notification (ECN) marking is observed at any time during
standard slow start or CSS, enter congestion avoidance
by setting the ssthresh to the current cwnd.
ssthresh = cwnd
Tuning Constants and Other Considerations It is RECOMMENDED that a HyStart++ implementation use
the following constants:
MIN_RTT_THRESH = 4 msec
MAX_RTT_THRESH = 16 msec
MIN_RTT_DIVISOR = 8
N_RTT_SAMPLE = 8
CSS_GROWTH_DIVISOR = 4
CSS_ROUNDS = 5
L = infinity if paced, L = 8 if non-paced
These constants have been determined with lab measurements
and real-world deployments. An implementation MAY tune them for
different network characteristics.
The delay increase sensitivity is determined
by MIN_RTT_THRESH and MAX_RTT_THRESH. Smaller values of
MIN_RTT_THRESH may cause spurious exits from slow start. Larger
values of MAX_RTT_THRESH may result in slow start not exiting
until loss is encountered for connections on large RTT paths.
MIN_RTT_DIVISOR is a fraction of RTT to compute the delay threshold.
A smaller value would mean a larger threshold and thus less sensitivity to
delay increase, and vice versa.
While all TCP implementations are REQUIRED to take at least one RTT
sample each round, implementations of HyStart++ are RECOMMENDED to take
at least N_RTT_SAMPLE RTT samples. Using lower values of N_RTT_SAMPLE will
lower the accuracy of the measured RTT for the round;
higher values will improve accuracy at the cost of more
processing.
The minimum value of CSS_GROWTH_DIVISOR MUST be at least 2.
A value of 1 results in the same aggressive behavior as regular
slow start. Values larger than 4
will cause the algorithm to be less aggressive and maybe less
performant.
Smaller values of CSS_ROUNDS may miss detecting jitter,
and larger values may limit performance.
Packet pacing is a possible mechanism to
avoid large bursts and their associated harm. A paced TCP implementation SHOULD
use L = infinity. Burst concerns are mitigated by pacing, and this
setting allows for optimal cwnd growth on modern networks.
For TCP implementations that pace to mitigate burst concerns, L
values smaller than infinity may suffer performance problems due to slow
cwnd growth in high-speed networks. For non-paced TCP implementations, L values
smaller than 8 may suffer performance problems due to slow cwnd growth in high-speed networks; L values larger than 8 may cause an increase in burstiness
and thereby loss rates, and result in poor performance.
An implementation SHOULD use HyStart++ only for the
initial slow start (when the ssthresh is at its initial value
of arbitrarily high per ) and fall
back to using standard slow start for the remainder of
the connection lifetime. This is acceptable because subsequent
slow starts will use the discovered ssthresh value to exit slow
start and avoid the overshoot problem. An implementation MAY
use HyStart++ to grow the restart window
after a long idle period.
In application-limited scenarios, the amount of data in
flight could fall below the bandwidth-delay product (BDP) and
result in smaller RTT samples, which can trigger an exit back to
slow start. It is expected that a connection might oscillate
between CSS and slow start in such scenarios. But this behavior
will neither result in a connection prematurely entering
congestion avoidance nor cause overshooting compared to
slow start.
Deployments and Performance Evaluations At the time of this writing, HyStart++ as described
in this document has been default enabled for all TCP
connections in the Windows operating system for over
two years with pacing disabled and an actual L = 8.
In lab measurements with Windows TCP, HyStart++ shows
goodput improvements as well as reductions in packet
loss and retransmissions compared to standard slow start.
For example, across a variety of tests on a 100 Mbps link
with a bottleneck buffer size of bandwidth-delay product,
HyStart++ reduces bytes retransmitted by 50% and
retransmission timeouts (RTOs) by 36%.
In an A/B test where we compared an implementation of HyStart++
(based on an earlier draft version of this document) to
standard slow start across a large Windows device
population, out of 52 billion TCP connections, 0.7% of
connections move from 1 RTO to 0 RTOs and another 0.7% of
connections move from 2 RTOs to 1 RTO with HyStart++.
This test did not focus on send-heavy connections, and
the impact on send-heavy connections is likely much
higher. We plan to conduct more such production
experiments to gather more data in the future.
Security Considerations HyStart++ enhances slow start and inherits the general
security considerations discussed in .
An attacker can cause HyStart++ to exit slow start prematurely
and impair the performance of a TCP connection by, for example,
dropping data packets or their acknowledgments.The ACK division attack outlined in does not affect
HyStart++ because the congestion window increase in HyStart++ is based
on the number of bytes newly acknowledged in each arriving ACK rather than by
a particular constant on each arriving ACK.
IANA ConsiderationsThis document has no IANA actions.ReferencesNormative ReferencesKey words for use in RFCs to Indicate Requirement LevelsIn many standards track documents several words are used to signify the requirements in the specification. These words are often capitalized. This document defines these words as they should be interpreted in IETF documents. This document specifies an Internet Best Current Practices for the Internet Community, and requests discussion and suggestions for improvements.TCP Congestion ControlThis document defines TCP's four intertwined congestion control algorithms: slow start, congestion avoidance, fast retransmit, and fast recovery. In addition, the document specifies how TCP should begin transmission after a relatively long idle period, as well as discussing various acknowledgment generation methods. This document obsoletes RFC 2581. [STANDARDS-TRACK]Ambiguity of Uppercase vs Lowercase in RFC 2119 Key WordsRFC 2119 specifies common key words that may be used in protocol specifications. This document aims to reduce the ambiguity by clarifying that only UPPERCASE usage of the key words have the defined special meanings.Informative ReferencesUnderstanding the performance of TCP pacingProceedings IEEE INFOCOM 2000Taming the elephants: New TCP slow startComputer Networks vol. 55, no. 9, pp. 2092-2110Requirements for Internet Hosts - Communication LayersThis RFC is an official specification for the Internet community. It incorporates by reference, amends, corrects, and supplements the primary protocol standards documents relating to hosts. [STANDARDS-TRACK]Path MTU discoveryThis memo describes a technique for dynamically discovering the maximum transmission unit (MTU) of an arbitrary internet path. It specifies a small change to the way routers generate one type of ICMP message. For a path that passes through a router that has not been so changed, this technique might not discover the correct Path MTU, but it will always choose a Path MTU as accurate as, and in many cases more accurate than, the Path MTU that would be chosen by current practice. [STANDARDS-TRACK]Packetization Layer Path MTU DiscoveryThis document describes a robust method for Path MTU Discovery (PMTUD) that relies on TCP or some other Packetization Layer to probe an Internet path with progressively larger packets. This method is described as an extension to RFC 1191 and RFC 1981, which specify ICMP-based Path MTU Discovery for IP versions 4 and 6, respectively. [STANDARDS-TRACK]QUIC Loss Detection and Congestion ControlThis document describes loss detection and congestion control mechanisms for QUIC.Stream Control Transmission ProtocolThis document describes the Stream Control Transmission Protocol (SCTP) and obsoletes RFC 4960. It incorporates the specification of the chunk flags registry from RFC 6096 and the specification of the I bit of DATA chunks from RFC 7053. Therefore, RFCs 6096 and 7053 are also obsoleted by this document. In addition, RFCs 4460 and 8540, which describe errata for SCTP, are obsoleted by this document.SCTP was originally designed to transport Public Switched Telephone Network (PSTN) signaling messages over IP networks. It is also suited to be used for other applications, for example, WebRTC.SCTP is a reliable transport protocol operating on top of a connectionless packet network, such as IP. It offers the following services to its users:The design of SCTP includes appropriate congestion avoidance behavior and resistance to flooding and masquerade attacks.TCP congestion control with a misbehaving receiverACM SIGCOMM Computer Communication Review, vol. 29, issue 5, pp. 71-78Acknowledgments During the discussions of this work on the TCPM mailing list and in working group meetings,
helpful comments, critiques, and reviews were received from (listed alphabetically by last name)
, , , , , , ,
, , , , and .
Authors' AddressesConfluent899 West Evelyn AveMountain ViewCA94041United States of Americapravb.ietf@gmail.comMicrosoftOne Microsoft WayRedmondWA98052United States of America+1 425 703 0447huanyi@microsoft.comMicrosoftOne Microsoft WayRedmondWA98052United States of America+1 425 538 8598maolson@microsoft.com