Subject: [tcpm] TCP sendbuffer advertising

From: Costin Raiciu <[email-masked]>
Date: Mon, 13 Jul 2015 23:24:46 +0300

Dear all,

We have been working on a simple extension to TCP that encode the amount of backlogged bytes in each TCP segment. This information can be used by the network or the receiver for various optimizations.

I will be present our work during the TCPM working group in Prague. This email is a heads-up for those interested in hearing more about the work before it is presented.

We have written a draft on the encoding, but unfortunately we have missed the Prague cutoff. You can download the draft from my webpage at http://nets.cs.pub.ro/~costin/files/draft-agache-tcpm-sndbufadv.txt
We will submit it properly as soon as the I-D submission tool reopens on saturday.

We also have a workshop paper in HotClout 2015 on the various use-cases of send buffer advertising, and our prototype implementation, that you can find at: http://nets.cs.pub.ro/~costin/files/sndbuf.pdf

We are looking forward to your comments.

Best.
Costin
----------------------------------------------
Costin Raiciu
Associate Professor
Homepage: http://nets.cs.pub.ro/~costin/
CS Department
University Politehnica of Bucharest

---

From: Yuchung Cheng <[email-masked]>
Date: Tue, 14 Jul 2015 09:51:29 -0700

> We are looking forward to your comments.
Some quick thoughts:

1) what happens when the backlog / sendbuf > 65536B?

2) announcing the backlog may help larger incast when the response >
cwnd (i.e., taking multiple round trips to deliver). but many incast
patterns I know of fit entirely in the first cwnd. For example 1000
senders write 2 packets each to a single receiver. ECN/dctcp has the
same limitation on such type of incast.

3) some application prefer late-binding and delays writing data into
the socket with features like TCP_NOTSENT_LOWAT
(https://lwn.net/Articles/560082/). Thus the sendbuf remains low while
app backlog may be very large. The react function that uses the
sendbuf info should not assume sendbuf == app backlog.

4) for HTTP/2 often the conversation is bi-directional: the server may
be responding to a HTTP/2 request while actively acking other new
client requests.
The receiver window of the packets will alternate between sendbuf or
receiver window. hence the sendbuf info may be missing intermittently.

but I think this is an interesting feature worth exploring!

> _______________________________________________
> tcpm mailing list
> [email-masked]
> https://www.ietf.org/mailman/listinfo/tcpm

---

From: Alejandro Popovsky <[email-masked]>
Date: Thu, 16 Jul 2015 20:28:15 -0300

Hi Costin,

Sender buffer info would be useful to check if the flow is network limited
(NL), data-generation limited (DG), or flow control limited (FC). But a one
bit field might be enough. I'll comment the draft below:

> A key information needed by networks is whether a connection is
> limited by the network or by the application.  This information
> is very difficult to accurately infer by passive measurements.

We are currently doing that without this extra buffer info for all 
connections
between ISP's flowing through the IXP's of the Argentine Chamber of Internet
(CABASE), by following all connections state using UpPerfomanceAnalyzer. 
This
analysis is used to evaluate real time end-user performance and end-to-end
capacity.

> This information enables network boxes and receivers to easily
> identify connections that need more capacity....

It might be dangerous to use it to assign capacity in network boxes,
because end users could just lie about their backlogged data in order to 
steal
capacity from neighbor connections.

> ...to estimate the sender congestion window and to accurately
> monitor flight-size....

> ...having SND.NXT smaller than WRITE.SEQ indicates that the
> congestion window is not large enough, so the connection is
> network limited at that point in time.

It is not absolutely necessary to estimate sender congestion window size 
in order
to check Network Limitation (NL). When senders hit their congestion window
top, they become ack-clocked and this can be detected at the monitor point
by checking both the data and ack flow timings.

Filling the congestion window is also not necessarily an indication of
Network Limitation (NL), because the congestion window may be smaller
than the current bandwidth delay product at the time. In such case the 
sender
is ack clocked but the data segment pace is still a burst per round trip 
time.
Anyhow, it would simplify the limitation analysis to have a bit at the 
end of
each burst indicating if transmission was stopped by the window or by the
lack of data to send.

> As long as the receive window is not a bottleneck...

Filling the receiver window is again not always an indication of flow 
control
limitation (FC) limitation. If receiver window is bigger than the bandwidth
delay product (BDP), the sender can still reach the available capacity. 
It can
be observed in some of the chamber ISP's with old tcp stacks in their
cache-boxes, with many connections reaching their available capacity
even with their rcvd window full.

All this said, including info about the sender buffer looks positive,
and would help limitation analysis to be less demanding.  Although
one bit field might suffice, something similar to the push bit,
or a small change to it.

Best regards, Alejandro Popovsky.

---

From: Costin Raiciu <[email-masked]>
Date: Fri, 17 Jul 2015 12:55:33 +0300

> bit field might be enough. I'll comment the draft below:

For most scenarios, a one bit field is enough, I agree. There are however some use cases where knowing the amount of backlogged data might help (e.g. in our HotCloud paper we can infer losses 
even if we sample packets 1:100 just by looking at cwnd information!).

> We are currently doing that without this extra buffer info for all connections
> (CABASE), by following all connections state using UpPerfomanceAnalyzer. This

Cool, I’d like to hear more about the operation of this box - however it sounds pretty expensive :).

> because end users could just lie about their backlogged data in order to steal

You are spot on here - this is an open question.

> It is not absolutely necessary to estimate sender congestion window size in order

You are right, but I would think you need to be close to the source to accurately detect packets released by the ACK clock. 
Surely measurements become quite noisy when you are close to the receiver, ACKs can get lost, etc

Even so, this solution is rather expensive as it needs to see most (if not all packets in both directions) - am I right? We are proposing a cheaper, always on option.

> than the current bandwidth delay product at the time. In such case the sender
> is ack clocked but the data segment pace is still a burst per round trip time.
> Anyhow, it would simplify the limitation analysis to have a bit at the end of

Sure, but if you average the send buffer information across enough RTTs you can understand fairly accurately if the network is the limiting factor or the app.

> Filling the receiver window is again not always an indication of flow control
> delay product (BDP), the sender can still reach the available capacity. It can

Agreed.

> one bit field might suffice, something similar to the push bit,

Thanks a lot for your comments. 
Costin