]> University of Oslo

PO Box 1080 Blindern N-0316 Oslo Norway naeemk@ifi.uio.no

University of Oslo

PO Box 1080 Blindern N-0316 Oslo Norway michawe@ifi.uio.no

Netflix Inc.

garmitage@netflix.com

University of Aberdeen

School of Engineering, Fraser Noble Building AB24 3UE Aberdeen United Kingdom gorry@erg.abdn.ac.uk

Transport TCP Maintenance and Minor Extensions ecn, aqm, congestion control, tcp Active Queue Management (AQM) mechanisms allow for burst tolerance while enforcing short queues to minimise the time that packets spend enqueued at a bottleneck. This can cause noticeable performance degradation for TCP connections traversing such a bottleneck, especially if there are only a few flows or their bandwidth-delay product (BDP) is large. The reception of a Congestion Experienced (CE) Explicit Congestion Notification (ECN) mark indicates that an AQM mechanism is used at the bottleneck, and the bottleneck network queue is therefore likely to be short. Feedback of this signal allows the TCP sender-side ECN reaction in congestion avoidance to reduce the Congestion Window (cwnd) by a smaller amount than the congestion control algorithm's reaction to inferred packet loss. Therefore, this specification defines an experimental change to the TCP reaction specified in RFC 3168, as permitted by RFC 8311.

Introduction Explicit Congestion Notification (ECN) makes it possible for an Active Queue Management (AQM) mechanism to signal the presence of incipient congestion without necessarily incurring packet loss. This lets the network deliver some packets to an application that would have been dropped if the application or transport did not support ECN. This packet loss reduction is the most obvious benefit of ECN, but it is often relatively modest. Other benefits of deploying ECN have been documented in .

Dweeb

young excitable person who may mature into a Nerd or Geek

Hacker

a clever programmer

Nerd

technically bright but socially inept person

Dweeb

young excitable person who may mature into a Nerd or Geek

Hacker

a clever programmer

Nerd

technically bright but socially inept person

The rules for ECN were originally written to be very conservative, and they required the congestion control algorithms of ECN-Capable Transport (ECT) protocols to treat indications of congestion signalled by ECN exactly the same as they would treat an inferred packet loss . Research has demonstrated the benefits of reducing network delays that are caused by interaction of loss-based TCP congestion control and excessive buffering . This has led to the creation of AQM mechanisms like Proportional Integral Controller Enhanced (PIE) and Controlling Queue Delay (CoDel) , which prevent bloated queues that are common with unmanaged and excessively large buffers deployed across the Internet . The AQM mechanisms mentioned above aim to keep a sustained queue short while tolerating transient (short-term) packet bursts. However, currently used loss-based congestion control mechanisms are not always able to effectively utilise a bottleneck link where there are short queues. For example, a TCP sender using the Reno congestion control needs to be able to store at least an end-to-end bandwidth-delay product (BDP) worth of data at the bottleneck buffer if it is to maintain full path utilisation in the face of loss- induced reduction of the congestion window (cwnd) . This amount of buffering effectively doubles the amount of data that can be in flight and the maximum round-trip time (RTT) experienced by the TCP sender. Modern AQM mechanisms can use ECN to signal the early signs of impending queue buildup long before a tail-drop queue would be forced to resort to dropping packets. It is therefore appropriate for the transport protocol congestion control algorithm to have a more measured response when it receives an indication with an early warning of congestion after the remote endpoint receives an ECN CE-marked packet. Recognizing these changes in modern AQM practices, the strict requirement that ECN CE signals be treated identically to inferred packet loss has been relaxed . This document therefore defines a new sender-side-only congestion control response called "ABE" (Alternative Backoff with ECN). ABE improves TCP's average throughput when routers use AQM-controlled buffers that allow only for short queues.

Definitions The key words "MUST", "MUST NOT", "REQUIRED", "SHALL", "SHALL NOT", "SHOULD", "SHOULD NOT", "RECOMMENDED", "NOT RECOMMENDED", "MAY", and "OPTIONAL" in this document are to be interpreted as described in BCP 14 when, and only when, they appear in all capitals, as shown here.

Specification This specification changes the congestion control algorithm of an ECN-Capable TCP transport protocol by changing the TCP-sender response to feedback from the TCP receiver that indicates the reception of a CE-marked packet, i.e., receipt of a packet with the ECN-Echo flag (defined in ) set, following the process defined in . The TCP-sender response is currently specified in Section 6.1.2 of the ECN specification and has been slightly updated by and and Section 6 of to read as:

• The indication of congestion should be treated just as a congestion loss in non-ECN-Capable TCP. That is, the TCP source halves the congestion window "cwnd" and reduces the slow start threshold "ssthresh", unless otherwise specified by an Experimental RFC in the IETF document stream.

As permitted by RFC 8311, this document specifies a sender-side change to TCP where receipt of a packet with the ECN-Echo flag SHOULD trigger the TCP source to set the slow start threshold (ssthresh) to 0.8 times the FlightSize, with a lower bound of 2 * SMSS applied to the result (where SMSS stands for Sender Maximum Segment Size)). As in , the TCP sender also reduces the cwnd value to no more than the new ssthresh value. Section 6.1.2 of RFC 3168 provides guidance on setting a cwnd less than 2 * SMSS.

Choice of ABE Multiplier ABE decouples the reaction of a TCP sender to inferred packet loss from the indication of ECN-signalled congestion in the congestion avoidance phase. To achieve this, ABE uses a different scaling factor for Equation 4 in Section 3.1 of . The description respectively uses beta_{loss} and beta_{ecn} to refer to the multiplicative decrease factors applied in response to inferred packet loss, and in response to a receiver indicating ECN-signalled congestion. For non-ECN-enabled TCP connections, only beta_{loss} applies. In other words, in response to inferred packet loss:

• ssthresh = max (FlightSize * beta_{loss}, 2 * SMSS)

and in response to an indication of an ECN-signalled congestion:

• ssthresh = max (FlightSize * beta_{ecn}, 2 * SMSS)
• and
• cwnd = ssthresh
• (If ssthresh == 2 * SMSS, Section 6.1.2 of RFC 3168 provides guidance on setting a cwnd lower than 2 * SMSS.)

where FlightSize is the amount of outstanding data in the network, upper-bounded by the smaller of the sender's cwnd and the receiver's advertised window (rwnd) . The higher the values of beta_{loss} and beta_{ecn}, the less aggressive the response of any individual backoff event. The appropriate choice for beta_{loss} and beta_{ecn} values is a balancing act between path utilisation and draining the bottleneck queue. More aggressive backoff (smaller beta_*) risks the underutilisation of the path, while less-aggressive backoff (larger beta_*) can result in slower draining of the bottleneck queue. The Internet has already been running with at least two different beta_{loss} values for several years: the standard value is 0.5 , and the Linux implementation of CUBIC has used a multiplier of 0.7 since kernel version 2.6.25 released in 2008. ABE does not change the value of beta_{loss} used by current TCP implementations. The recommendation in this document specifies a value of beta_{ecn}=0.8. This recommended beta_{ecn} value is only applicable for the standard TCP congestion control . The selection of beta_{ecn} enables tuning the response of a TCP connection to shallow AQM-marking thresholds. beta_{loss} characterizes the response of a congestion control algorithm to packet loss, i.e., exhaustion of buffers (of unknown depth). Different values for beta_{loss} have been suggested for TCP congestion control algorithms. Consequently, beta_{ecn} is likely to be an algorithm-specific parameter rather than a constant multiple of the algorithm's existing beta_{loss}. A range of tests (Section IV of ) with NewReno and CUBIC over CoDel and PIE in lightly multiplexed scenarios have explored this choice of parameter. The results of these tests indicate that CUBIC connections benefit from beta_{ecn} of 0.85 (cf. beta_{loss} = 0.7), and NewReno connections see improvements with beta_{ecn} in the range 0.7 to 0.85 (cf. beta_{loss} = 0.5).

Discussion Much of the technical background for ABE can be found in , which uses a mix of experiments, theory, and simulations with NewReno and CUBIC to evaluate its performance. ABE was shown to present significant performance gains in lightly-multiplexed (few concurrent flows) scenarios, without losing the delay-reduction benefits of deploying CoDel or PIE. The performance improvement is achieved when reacting to ECN-Echo in congestion avoidance (when ssthresh > cwnd) by multiplying cwnd and ssthresh with a value in the range [0.7,0.85]. Applying ABE when cwnd is smaller than or equal to ssthresh is not currently recommended, but its use in that scenario may benefit from additional attention, experimentation, and specification.

Rationale for Using ECN to Vary the Degree of Backoff AQM mechanisms such as CoDel and PIE set a delay target in routers and use congestion notifications to constrain the queuing delays experienced by packets rather than in response to impending or actual bottleneck buffer exhaustion. With current default delay targets, CoDel and PIE both effectively emulate a bottleneck with a short queue (Section II of ) while also allowing short traffic bursts into the queue. This provides acceptable performance for TCP connections over a path with a low BDP, or in highly multiplexed scenarios (many concurrent transport flows). However, in a lightly multiplexed case over a path with a large BDP, conventional TCP backoff leads to gaps in packet transmission and underutilisation of the path. Instead of discarding packets, an AQM mechanism is allowed to mark ECN-Capable packets with an ECN CE mark. The reception of CE-mark feedback not only indicates congestion on the network path, it also indicates that an AQM mechanism exists at the bottleneck along the path. Therefore, the CE mark likely came from a bottleneck with a controlled short queue. Reacting differently to an ECN-signalled congestion than to an inferred packet loss can then yield the benefit of a reduced backoff when queues are short. Using ECN can also be advantageous for several other reasons . The idea of reacting differently to inferred packet loss and detection of an ECN-signalled congestion predates this specification, e.g., previous research proposed using ECN CE-marked feedback to modify TCP congestion control behaviour via a larger multiplicative decrease factor in conjunction with a smaller additive increase factor . The goal of this former work was to operate across AQM bottlenecks (using Random Early Detection (RED)) that were not necessarily configured to emulate a short queue. (The current usage of RED as an Internet AQM method is limited .)

An RTT-Based Response to Indicated Congestion This specification applies to the use of ECN feedback as defined in , which specifies a response to indicated congestion that is no more frequent than once per path round-trip time. Since ABE responds to indicated congestion once per RTT, it does not respond to any further loss within the same RTT because an ABE sender has already reduced the congestion window. If congestion persists after such reduction, ABE continues to reduce the congestion window in each consecutive RTT. This consecutive reduction can protect the network against long-standing unfairness in the case of AQM algorithms that do not keep a small average queue length. The mechanism does not rely on Accurate ECN . In contrast, transport protocol mechanisms can also be designed to utilise more frequent and detailed ECN feedback (e.g., Accurate ECN ), which then permit a congestion control response that adjusts the sending rate more frequently. Data Center TCP (DCTCP) is an example of this approach.

ABE Deployment Requirements This update is a sender-side-only change. Like other changes to congestion control algorithms, it does not require any change to the TCP receiver or to network devices. It does not require any ABE-specific changes in routers or the use of Accurate ECN feedback by a receiver. If the method is only deployed by some senders, and not by others, the senders using it can gain some advantage, possibly at the expense of other flows that do not use this updated method. Because this advantage applies only to ECN-marked packets and not to packet-loss indications, an ECN-Capable bottleneck will still fall back to dropping packets if a TCP sender using ABE is too aggressive. The result is no different than if the TCP sender were using traditional loss-based congestion control. When used with bottlenecks that do not support ECN marking, the specification does not modify the transport protocol.

ABE Experiment Goals states that the congestion control response following an indication of ECN-signalled congestion is the same as the response to a dropped packet. updates this specification to allow systems to provide a different behaviour when they experience ECN-signalled congestion rather than packet loss. The present specification defines such an experiment and is an Experimental RFC. We expect to propose it as a Standards-Track document in the future. The purpose of the Internet experiment is to collect experience with the deployment of ABE and confirm acceptable safety in deployed networks that use this update to TCP congestion control. To evaluate ABE, this experiment requires support in AQM routers for the ECN-marking of packets carrying the ECN-Capable Transport codepoint ECT(0) . The result of this Internet experiment ought to include an investigation of the implications of experiencing an ECN-CE mark followed by loss within the same RTT. At the end of the experiment, this will be reported to the TCPM Working Group or the IESG. ABE is implemented as a patch for Linux and FreeBSD. This is meant for research and experimentation and is available for download at <https://heim.ifi.uio.no/michawe/research/abe/>. This code was used to produce the test results that are reported in . The FreeBSD code was committed to the mainline kernel on March 19, 2018 .

IANA Considerations This document has no IANA actions.

Security Considerations The described method is a sender-side-only transport change, and it does not change the protocol messages exchanged. Therefore, the security considerations for ECN still apply. This is a change to TCP congestion control with ECN that will typically lead to a change in the capacity achieved when flows share a network bottleneck. This could result in some flows receiving more than their fair share of capacity. Similar unfairness in the way that capacity is shared is also exhibited by other congestion control mechanisms that have been in use in the Internet for many years (e.g., CUBIC ). Unfairness may also be a result of other factors, including the round-trip time experienced by a flow. ABE applies only when ECN-marked packets are received, not when packets are lost. Therefore, use of ABE cannot lead to congestion collapse.

References Normative References In many standards track documents several words are used to signify the requirements in the specification. These words are often capitalized. This document defines these words as they should be interpreted in IETF documents. This document specifies an Internet Best Current Practices for the Internet Community, and requests discussion and suggestions for improvements. This memo specifies the incorporation of ECN (Explicit Congestion Notification) to TCP and IP, including ECN's use of two bits in the IP header. [STANDARDS-TRACK] This document defines TCP's four intertwined congestion control algorithms: slow start, congestion avoidance, fast retransmit, and fast recovery. In addition, the document specifies how TCP should begin transmission after a relatively long idle period, as well as discussing various acknowledgment generation methods. This document obsoletes RFC 2581. [STANDARDS-TRACK] This memo presents recommendations to the Internet community concerning measures to improve and preserve Internet performance. It presents a strong recommendation for testing, standardization, and widespread deployment of active queue management (AQM) in network devices to improve the performance of today's Internet. It also urges a concerted effort of research, measurement, and ultimate deployment of AQM mechanisms to protect the Internet from flows that are not sufficiently responsive to congestion notification. Based on 15 years of experience and new research, this document replaces the recommendations of RFC 2309. This Informational RFC describes Data Center TCP (DCTCP): a TCP congestion control scheme for data-center traffic. DCTCP extends the Explicit Congestion Notification (ECN) processing to estimate the fraction of bytes that encounter congestion rather than simply detecting that some congestion has occurred. DCTCP then scales the TCP congestion window based on this estimate. This method achieves high-burst tolerance, low latency, and high throughput with shallow- buffered switches. This memo also discusses deployment issues related to the coexistence of DCTCP and conventional TCP, discusses the lack of a negotiating mechanism between sender and receiver, and presents some possible mitigations. This memo documents DCTCP as currently implemented by several major operating systems. DCTCP, as described in this specification, is applicable to deployments in controlled environments like data centers, but it must not be deployed over the public Internet without additional measures. This memo updates RFC 3168, which specifies Explicit Congestion Notification (ECN) as an alternative to packet drops for indicating network congestion to endpoints. It relaxes restrictions in RFC 3168 that hinder experimentation towards benefits beyond just removal of loss. This memo summarizes the anticipated areas of experimentation and updates RFC 3168 to enable experimentation in these areas. An Experimental RFC in the IETF document stream is required to take advantage of any of these enabling updates. In addition, this memo makes related updates to the ECN specifications for RTP in RFC 6679 and for the Datagram Congestion Control Protocol (DCCP) in RFCs 4341, 4342, and 5622. This memo also records the conclusion of the ECN nonce experiment in RFC 3540 and provides the rationale for reclassification of RFC 3540 from Experimental to Historic; this reclassification enables new experimental use of the ECT(1) codepoint. RFC 2119 specifies common key words that may be used in protocol specifications. This document aims to reduce the ambiguity by clarifying that only UPPERCASE usage of the key words have the defined special meanings. Informative References CUBIC is an extension to the current TCP standards. It differs from the current TCP standards only in the congestion control algorithm on the sender side. In particular, it uses a cubic function instead of a linear window increase function of the current TCP standards to improve scalability and stability under fast and long-distance networks. CUBIC and its predecessor algorithm have been adopted as defaults by Linux and have been used for many years. This document provides a specification of CUBIC to enable third-party implementations and to solicit community feedback through experimentation on the performance of CUBIC. The goal of this document is to describe the potential benefits of applications using a transport that enables Explicit Congestion Notification (ECN). The document outlines the principal gains in terms of increased throughput, reduced delay, and other benefits when ECN is used over a network path that includes equipment that supports Congestion Experienced (CE) marking. It also discusses challenges for successful deployment of ECN. It does not propose new algorithms to use ECN nor does it describe the details of implementation of ECN in endpoint devices (Internet hosts), routers, or other network devices. Bufferbloat is a phenomenon in which excess buffers in the network cause high latency and latency variation. As more and more interactive applications (e.g., voice over IP, real-time video streaming, and financial transactions) run in the Internet, high latency and latency variation degrade application performance. There is a pressing need to design intelligent queue management schemes that can control latency and latency variation, and hence provide desirable quality of service to users. This document presents a lightweight active queue management design called "PIE" (Proportional Integral controller Enhanced) that can effectively control the average queuing latency to a target value. Simulation results, theoretical analysis, and Linux testbed results have shown that PIE can ensure low latency and achieve high link utilization under various congestion situations. The design does not require per-packet timestamps, so it incurs very little overhead and is simple enough to implement in both hardware and software. This document describes CoDel (Controlled Delay) -- a general framework that controls bufferbloat-generated excess delay in modern networking environments. CoDel consists of an estimator, a setpoint, and a control loop. It requires no configuration in normal Internet deployments.

Acknowledgements Authors N. Khademi, M. Welzl, and G. Fairhurst were partly funded by the European Community under its Seventh Framework Programme through the Reducing Internet Transport Latency (RITE) project (ICT-317700). The views expressed are solely those of the authors. Author G. Armitage performed most of his work on this document while employed by Swinburne University of Technology, Melbourne, Australia. The authors would like to thank Stuart Cheshire for many suggestions when revising this document. They would also like to thank the following people for their contributions to : Chamil Kulatunga, David Ros, Stein Gjessing, and Sebastian Zander. Thanks also to (in alphabetical order) David Black, Roland Bless, Bob Briscoe, Markku Kojo, John Leslie, Lawrence Stewart, and the TCPM Working Group for providing valuable feedback on this document. Finally, the authors would like to thank everyone who provided feedback on the congestion control behaviour specified in this document that was received from the IRTF Internet Congestion Control Research Group (ICCRG).