Subject: [tcpm] rfc2581bis-01 and RFC 2861

From: Salman Abdul Baset <[email-masked]>
Date: Sat, 22 Jul 2006 14:02:43 -0400 (EDT)

(Note: This post is being sent to tcpm and end2end-interest mailing lists)

The issue is that RFC 2861 (Experimental) suggests to cap cwnd if application
is rate limited. For example, for sending CBR over TCP with RTT of 100ms
and CBR rate of 10ms, the cwnd should not theoretically increase beyond 11.

The latest TCP congestion control draft:

http://www.ietf.org/internet-drafts/draft-ietf-tcpm-rfc2581bis-01.txt

does not allude to RFC 2861. This latest draft perhaps tries to combine
the RFC 2581 (TCP Congestion Control) and RFC 3465 (Appropriate Byte Couting) in
one document. In $3.1, (page 5, par 5) it suggests:

"The RECOMMENDED way to increase cwnd during congestion avoidance is
    to count the number of bytes that have been acknowledged by ACKs for
    new data.  (A drawback of this implementation is that it requires
    maintaining an additional state variable.)  When the number of bytes
    acknowledged reaches cwnd, then cwnd can be incremented by up to
    SMSS bytes.  Note that during congestion avoidance, cwnd MUST NOT be
    increased by more than SMSS bytes per RTT.  This method both allows
    TCPs to increase cwnd by one segment per RTT in the face of delayed
    ACKs and provides robustness against ACK Division attacks."

It clearly states that cwnd should be increased if number of bytes in
current cwnd have been acknowledged. This implies that cwnd increase will
also happen for CBR traffic, albeit at a slower rate because of
application rate limitation.

I am not sure if it is an intentional omission by the authors but this
seems to have significant implications for sending CBR over TCP. For
example, delays incurred by CBR stream can be quite different because of
cwnd limitation (due to application rate).

Regards,
Salman

_______________________________________________
tcpm mailing list
[email-masked]
https://www1.ietf.org/mailman/listinfo/tcpm

---

From: Mark Allman <[email-masked]>
Date: Sun, 23 Jul 2006 22:06:15 -0400

> (Note: This post is being sent to tcpm and end2end-interest mailing

I have cut end2end off this reply as this seems to be a WG matter and we
probably do not need to clutter e2e.

> does not allude to RFC 2861. This latest draft perhaps tries to
> combine the RFC 2581 (TCP Congestion Control) and RFC 3465
> (Appropriate Byte Couting) in one document. 

This is incorrect.  The goal of the above listed I-D is to fix problems
with the current congestion control *specification* (not algorithmic
problems).  And, further to roll in a couple of quite minor *standards
track* extensions to TCP (namely a larger initial cwnd (RFC 3390) and
limited transmit (RFC 3042).

We explicitly *do not* roll in Appropriate Byte Counting (RFC 3465)
because, as you note, it is not a standards track extension at this
time.

> In $3.1, (page 5, par 5) it suggests:
> 

Note that this is only a slight change from RFC 2581 and the above is
*not* ABC.  First, ABC is about how to increase cwnd during *slow start*
while the above paragraph discusses congestion avoidance.  Second, the
byte counting method for increasing cwnd during congestion avoidance is
specifically allowed in RFC 2581.  The change is that it is RECOMMENDED
in the latest draft.  (The change is an attempt to mitigate a security
problem with receiver's ACKing each byte in a packet separately to try
to artificially inflate the sender's cwnd.)

> example, delays incurred by CBR stream can be quite different because
> of cwnd limitation (due to application rate).

I do not follow the problem you are getting at here.  If you traffic is
CBR and you have enough cwnd to send at that CBR then why would
increasing it beyond what will provide the given rate impact the
traffic?  I just can't see the point you're trying to make here, can you
try again?

allman

---

From: Salman Abdul Baset <[email-masked]>
Date: Mon, 24 Jul 2006 01:17:17 -0400 (EDT)

> limited transmit (RFC 3042).

True. I guess the question is when/if RFC 2861 will be incorporated as a
Standards track rfc. As you are probably aware, Linux 2.4 and onwards
implement 2861. Ofcourse, this is not the reason for making an experimental
RFC a standard but I was hoping to find some effort towards it. Perhaps,
there is one but I have not scanned all the archives. Also, such an effort
is not evident from the WG description and 'Goals and Milestones'.

> traffic?  I just can't see the point you're trying to make here, can you

So I guess, there are two ways of increasing cwnd during CA. One is to
increase the window based on number of ACKs received (which is what Linux
is doing and eq (2) in RFC 2581) and the other is to increase it based on
number of bytes ACKed (2581: page 5, par 3). It appears that RFC 2581
recommends both. As you mentioned, the new draft recommends the later.

Now consider CBR over TCP. The window will increase quickly for the first
case assuming packets are smaller than MSS (100 byte packet and 1448 byte
MSS). It will increase slowly for the second case. For a MSS of 200 bytes
and packet size of 100 bytes, the cwnd increase rate is 1/2 of the first
case.

When there is a TDACK loss, cwnd will be reduced to half. The CBR over
TCP traffic is not impacted if after a loss cwnd is big enough to
maintain the rate.

Now consider RFC 2861. It says that cap the cwnd if application is
rate-limited. For a CBR rate of 100 packet/s (or every 10ms) and RTT of
100ms, the cwnd should not theoretically increase beyond 11 because the
number of packets inflight are 10. In practice, it increases to a higher
value (and stays there if there is no loss) because the packets are
buffered when cwnd < CBR rate.

A CBR over a TCP implementation that follows RFC 2861 is more likely to
suffer from cwnd limited delays than a TCP implementation that keeps
increasing cwnd.

Similarly, CBR over TCP also benefits from not doing byte counting during
slow start and CA.

Another interesting nuance is if cwnd is significantly higher than the
current traffic rate, shall it be slowly brought down? This phenomena is
different from the idle period reduction.

What do you think?

Regards,
Salman

---

From: Mark Allman <[email-masked]>
Date: Mon, 24 Jul 2006 07:36:22 -0400

There are a lot of disparate things in your email.  Let's see ...

> True. I guess the question is when/if RFC 2861 will be incorporated as
> a Standards track rfc. As you are probably aware, Linux 2.4 and
> onwards implement 2861. Ofcourse, this is not the reason for making an
> experimental RFC a standard but I was hoping to find some effort
> towards it. Perhaps, there is one but I have not scanned all the
> archives. Also, such an effort is not evident from the WG description
> and 'Goals and Milestones'.

With my WG chair hat on .... This WG certainly can be a forum for
advancing specs (e.g., from Experimental to PS).  That general notion is
in the charter and if someone wanted to bring forth the energy to
consider the advancing of RFC2861 I think that'd be fine for the WG to
consider.  Enough with the hat.

> increase the window based on number of ACKs received (which is what
> Linux is doing and eq (2) in RFC 2581) and the other is to increase it
> based on number of bytes ACKed (2581: page 5, par 3). It appears that
> RFC 2581 recommends both. As you mentioned, the new draft recommends
> the later.

The draft (and RFC 2581) *allows* both and the draft *recommends* byte
counting during CA.

> Now consider CBR over TCP. The window will increase quickly for the
> first case assuming packets are smaller than MSS (100 byte packet and
> 1448 byte MSS). It will increase slowly for the second case. For a MSS
> of 200 bytes and packet size of 100 bytes, the cwnd increase rate is
> 1/2 of the first case.

Yes.  One could argue that the faster increase (the traditional
approach) is inappropriate, but it is a pretty small effect (and, it can
cut both ways---sometimes being beneficial and sometimes not).

> TCP traffic is not impacted if after a loss cwnd is big enough to

RFC 2581 bases the reduction of cwnd on FlightSize (amount of
outstanding data) not cwnd.  So, the CBR over TCP will be reduced by
half.  I.e., if you have 10 packets outstanding, a cwnd of 100000
packets and take a loss your new cwnd should be 5 packets.  The notion
of FlightSize was introduced into the spec. to accommodate cases whereby
the cwnd does not accurately reflect the load being placed on the path.
(As has been discussed on this list before, this is not an ideal fix
because it causes issues in another case.)

allman

---

From: "Christian Huitema" <[email-masked]>
Date: Mon, 24 Jul 2006 09:35:18 -0700

> of FlightSize was introduced into the spec. to accommodate cases
> whereby the cwnd does not accurately reflect the load being placed on
> the path.

One may in fact question whether this is even the right thing. Suppose a
1 Mbps link is shared between a 64 kpbs "CBR" flow and a "best effort"
file transfer. The file transfer window will increase until it reaches
something larger than 1 Mbps. A that point, queues will build up, and
eventually some packets will be dropped. You certainly want the file
transfer rate to drop to 512 kbps, and then progressively increase. But
do you really want the CBR flow to drop to 32 kbps? Is that fair? Is
that even useful?

-- Christian Huitema

---

From: Salman Abdul Baset <[email-masked]>
Date: Mon, 24 Jul 2006 12:51:20 -0400 (EDT)

Thanks a lot for your comments. My comments inline.

> The draft (and RFC 2581) *allows* both and the draft *recommends* byte

True. I actually meant that but you put it the right way.

> cut both ways---sometimes being beneficial and sometimes not).

Can you elaborate how it may not be beneficial?

> because it causes issues in another case.)

You are right. However, as Christian Huitema pointed out, there are
fairness issues here.

Somewhere, I read about sharing information between different TCP
connections. That might be useful in this scenario to maintain fairness.

So an application like Skype when forced to send voice over a TCP
implementation (assuming UDP is blocked) which:

1) does ACK counting during CA
May increase the packet rate to increase the inflight packets to mitigate
the cwnd=ssthresh=(flightsize/2)

2) byte counting during CA
May increase the packet rate and voice packet size to MSS to mitigate the
cwnd decrease.

Regards,
Salman

---

From: Mark Allman <[email-masked]>
Date: Mon, 24 Jul 2006 12:52:15 -0400

> > packets and take a loss your new cwnd should be 5 packets.  The
> > notion of FlightSize was introduced into the spec. to accommodate
> > cases whereby the cwnd does not accurately reflect the load being
> > placed on the path.  (As has been discussed on this list before,
> > this is not an ideal fix because it causes issues in another case.)
> 
> One may in fact question whether this is even the right thing.

Certainly one could ask that question in general.  But, in the context
of the current discussion I don't know why one would.

My scenario above is: 10 packets outstanding + cwnd=100000 packets + one
  packet loss, which yields a cwnd of 5 packets (per RFC 2581)

Salman's note suggests: 10 packets outstanding + cwnd=10 packets
  (because of CWV) + one packet loss, which yields a cwnd of 5 packets
  (per RFC 2581)

I.e., both these cases yield the same thing---an appropriate reduction
of cwnd in the face inferred congestion.

I don't see how to really solve this problem you note in an endpoint
transport protocol without help from the congested point in the network.
One would hope that some smart AQM could drop in proportion to the
capacity a particular flow is using.

allman

---

From: Mark Allman <[email-masked]>
Date: Mon, 24 Jul 2006 13:12:40 -0400

> > approach) is inappropriate, but it is a pretty small effect (and, it
> > can cut both ways---sometimes being beneficial and sometimes not).
> 

If you increase too fast you may cause (more) congestion.

The ACK splitting attack is sort of the bounding example.  If the
receiver ACKs every byte then it sort of defeats congestion control.
E.g., by getting the sender to increase cwnd from one packet of 1460
bytes to 1460 packets of 1460 bytes each (over 2MB) in one RTT.  That
sort of increase rate is not consistent with the spirit of slow start
and could certainly cause congestion on the network.

In congestion avoidance which way you increase things really doesn't
matter as much because the increase is linear and while ACKing every
byte still might help, it is probably not going to cause a big problem.

> You are right. However, as Christian Huitema pointed out, there are

See my response.

Fairness depends on what you are competing with.  Christian's example is
a fine example of inequity.  But, how does an endhost tell the
difference between his scenario and a scenario whereby a zillion flows
are all sending at 10 packets/second and therefore when there is loss
the flow in question ought to be throttled?  No amount of counting of
bytes or using CWV of cwnd vs. FlightSize is going to give the flow the
information to make the right decision.

allman

---

From: Salman Abdul Baset <[email-masked]>
Date: Mon, 24 Jul 2006 14:55:28 -0400 (EDT)

> If you increase too fast you may cause (more) congestion.

Perhaps, it should not have a large (or perhaps even a small) effect on
congestion because the application is CBR over TCP i.e. the application
is rate-limited. For voice, the max rate is 32 kb/s (one way) or less.

> byte still might help, it is probably not going to cause a big problem.

In the ACK counting case for increasing cwnd, the question is whether
ACKing every byte will increase the cwnd quickly enough to minimize the
sender incurred delay due to cwnd limitation for a CBR over TCP flow.

> information to make the right decision.

True. An endhost cannot tell the difference between the scenarios you
mentioned.

I think an interesting question is how two flows, one FTP and one CBR
over TCP originating from the same host and destined for the same host
(src!=dest) compete for bandwidth. My guess is that eventually, FTP flow
will eat into the bandwidth of CBR over TCP flow. I can probably check
that for a FTP flow competing with a Skype voice over TCP flow in a
controlled environment.

Generalizing further, how large FTP-like TCP flows have an impact on small
application bandwidth/rate limited TCP flows. I have not completely
scanned the literature to comment if anyone has already analyzed it.

Regards,
Salman