Subject: [tcpm] I-D Action: draft-ietf-tcpm-newcwv-02.txt

From: [email-masked]
Date: Mon, 01 Jul 2013 12:49:25 -0700

A New Internet-Draft is available from the on-line Internet-Drafts directories.
 This draft is a work item of the TCP Maintenance and Minor Extensions Working Group of the IETF.

	Title           : Updating TCP to support Rate-Limited Traffic
	Author(s)       : Godred Fairhurst
                          Arjuna Sathiaseelan
                          Raffaello Secchi
	Filename        : draft-ietf-tcpm-newcwv-02.txt
	Pages           : 19
	Date            : 2013-07-01

Abstract:
   This document proposes an update to RFC 5681 to address issues that
   arise when TCP is used to support traffic that exhibits periods where
   the sending rate is limited by the application rather than the
   congestion window.  It updates TCP to allow a TCP sender to restart
   quickly following either an idle or rate-limited interval.  This
   method is expected to benefit applications that send rate-limited
   traffic using TCP, while also providing an appropriate response if
   congestion is experienced.

   It also evaluates the Experimental specification of TCP Congestion
   Window Validation, CWV, defined in RFC 2861, and concludes that RFC
   2861 sought to address important issues, but failed to deliver a
   widely used solution.  This document therefore recommends that the
   status of RFC 2861 is moved from Experimental to Historic, and that
   it is replaced by the current specification.

   NOTE: The standards status of this WG document is under review for
   consideration as either Experimental (EXP) or Proposed Standard (PS).
   This decision will be made later as the document is finalised.

The IETF datatracker status page for this draft is:
https://datatracker.ietf.org/doc/draft-ietf-tcpm-newcwv

There's also a htmlized version available at:
http://tools.ietf.org/html/draft-ietf-tcpm-newcwv-02

A diff from the previous version is available at:
http://www.ietf.org/rfcdiff?url2=draft-ietf-tcpm-newcwv-02

Internet-Drafts are also available by anonymous FTP at:
ftp://ftp.ietf.org/internet-drafts/

---

From: [email-masked]
Date: Tue, 2 Jul 2013 08:20:06 +0100

We've just uploaded a new version of our draft. The congestion window
management algorithm has not changed since -00, but in -01 and -02 drafts
we introduce a more robust way to measure the pipeACK that attempts to
account for the wide variety of interactive TCP application behaviour.
This uses a sampling method to sample pipeACK over the last second.

Clearly the cwnd poorly reflects the capacity in the non-validated phase
and I think it would be wrong to use this for TCP control block sharing. 
We therefore also  introduce text that proposes that TCB sharing should
use the pipeACK in place of cwnd when a TCP sender is in the Nonvalidated
phase.  We think this value better reflects the capacity that the flow has
utilised in the network path - but would welcome more thought on this.
Thoughts on this would be most welcome.

Gorry

> _______________________________________________
> tcpm mailing list
> [email-masked]
> https://www.ietf.org/mailman/listinfo/tcpm
>

---

From: "Scharf, Michael (Michael)" <[email-masked]>
Date: Tue, 16 Jul 2013 09:38:30 +0000

Hi Gorry,

Regarding Section 4.4.1. Response to congestion in the nonvalidated phase:

>   At the end of the recovery phase, the TCP sender MUST reset the cwnd
>   using the method below:
>           cwnd = ((LossFlightSize - R)/2).
>
>   Where, R is the volume of data that was retransmitted during the
>   recovery phase.  This follows the method proposed for Jump Start
>   [Liu07].  The inclusion of the term R makes this adjustment more
>   conservative than standard TCP.  (This is required, since the sender
>   may have sent more segments than a Standard TCP sender would have
>   done.  The additional reduction is beneficial when the LossFlightSize
>   significantly overshoots the available path capacity incurring
>   significant loss, for instance an intense traffic burst following a
>   non-validated period.)

As already mentioned in http://www.ietf.org/mail-archive/web/tcpm/current/msg08015.html, when implementing [Liu07] in Linux some years ago, I ran into situations where R was larger than LossFlightSize (possibly due to multiple retransmissions of the same segment). The equation cwnd = ((LossFlightSize - R)/2) could thus result in a negative cwnd. That issue might partly depend on how to exactly R is measured, but I think that this equation requires further analysis and can't be mandated by a MUST.

Even if (LossFlightSize - R) is positive, I think that M. Allman's offlist comment about performance loss in loss of a short burst (Section 9.1) has to be addressed.

For instance, I wonder why cwnd at the end of recovery phase could not be reset to the value at the beginning, i.e.,

 cwnd = Min(cwnd/2,Max(pipeACK,LossFlightSize)) 

If that value is a "save" value for loss recovery, why is it not save to use it after loss recovery?

Thanks

Michael

> -----Original Message-----
> From: [email-masked] [mailto:[email-masked]] On 
> Behalf Of [email-masked]
> Sent: Tuesday, July 02, 2013 9:20 AM
> To: [email-masked]
> Subject: [tcpm] New revision: draft-ietf-tcpm-newcwv-02.txt
> 
> We've just uploaded a new version of our draft. The 
> congestion window management algorithm has not changed since 
> -00, but in -01 and -02 drafts we introduce a more robust way 
> to measure the pipeACK that attempts to account for the wide 
> variety of interactive TCP application behaviour.
> Clearly the cwnd poorly reflects the capacity in the 
> non-validated phase and I think it would be wrong to use this 
> for TCP control block sharing. 
> We therefore also  introduce text that proposes that TCB 
> sharing should use the pipeACK in place of cwnd when a TCP 
> sender is in the Nonvalidated phase.  We think this value 
> better reflects the capacity that the flow has utilised in 
> the network path - but would welcome more thought on this.
> >  This draft is a work item of the TCP Maintenance and Minor 
> Extensions 
> > Working Group of the IETF.
> >

---

From: [email-masked]
Date: Thu, 18 Jul 2013 07:38:40 +0100

Hi Michael,

I feel I should start by making sure we are on the same page with respect
to "R".
In CWV, we consider apps that do not always fill cwnd to probe for
capacity, hence there can be cases where a TCP flow can change its
behaviour. If we allow a cwnd that corresponds to much more than the
average sending rate (which we do), then I suggest we need to
significantly reduce the cwnd when there is a significant "mistake" in
sending more than the current path capacity. This is the role of "R".

Originally we thought "R" should be computed from SACK Information. That
is what we had simulated before, and I think follows part of Mark Allman's
JS work - but we now think this can be troublesome.   Instead, we now use
a method that counts retransmitted segments during the recovery phase for
the first window of data. We'd be very happy to talk about how we
currently do this, especially if you have experience (Raffaello can say
more, in cc).

See in-line for the second half of the answer.

Gorry

> As already mentioned in
> http://www.ietf.org/mail-archive/web/tcpm/current/msg08015.html, when
> implementing [Liu07] in Linux some years ago, I ran into situations where
> R was larger than LossFlightSize (possibly due to multiple retransmissions
> of the same segment). The equation cwnd = ((LossFlightSize - R)/2) could
> thus result in a negative cwnd. That issue might partly depend on how to
> exactly R is measured, but I think that this equation requires further
> analysis and can't be mandated by a MUST.
>
> Even if (LossFlightSize - R) is positive, I think that M. Allman's offlist
> comment about performance loss in loss of a short burst (Section 9.1) has
> to be addressed.
>
> For instance, I wonder why cwnd at the end of recovery phase could not be
> reset to the value at the beginning, i.e.,
>
> If that value is a "safe" value for loss recovery, why is it not save to
> use it after loss recovery?
>
I think this formula can't be used AFTER recovery.

This is why we chose to use this during recovery:

- At the start of loss recovery, the sender needs to react at least the
same as normal TCP, by halving cwnd. But: cwnd/2 is halving something that
was not validated, and did not provide any indication of the actual data
sent - in the NVP, the cwnd is only an upper limit.

The second term above reflects recently used capacity;

-  pipeACK is intended as a measure of capacity used in the recent past,
but it can be quite different to (D-R) in cases when the traffic pattern
changes inducing congestion. It helps though in not under-estimating
capacity.

- LossFlightSize (D) reflects the pattern that induced the congestion -
this is already in-flight during loss recovery.

The value of cwnd at the end of the loss recovery should not be dependent
on an initial cwnd that was not itself validated.

---

From: "Scharf, Michael (Michael)" <[email-masked]>
Date: Thu, 18 Jul 2013 14:12:44 +0000

Hi Gorry, 

> From: [email-masked] [mailto:[email-masked]] 
> Sent: Thursday, July 18, 2013 8:39 AM
> To: Scharf, Michael (Michael)
> Cc: [email-masked]; [email-masked]; [email-masked]
> Subject: RE: [tcpm] New revision: draft-ietf-tcpm-newcwv-02.txt
> 
> Hi Michael,
> 
> I feel I should start by making sure we are on the same page 
> with respect to "R".
> In CWV, we consider apps that do not always fill cwnd to 
> probe for capacity, hence there can be cases where a TCP flow 
> can change its behaviour. If we allow a cwnd that corresponds 
> to much more than the average sending rate (which we do), 
> then I suggest we need to significantly reduce the cwnd when 
> there is a significant "mistake" in sending more than the 
> current path capacity. This is the role of "R".
>
> Originally we thought "R" should be computed from SACK 
> Information. That is what we had simulated before, and I 
> think follows part of Mark Allman's
> a method that counts retransmitted segments during the 
> recovery phase for the first window of data. We'd be very 
> happy to talk about how we currently do this, especially if 
> you have experience (Raffaello can say more, in cc).

My main concern is that a substraction of two counters can be negative or 0, and this is not addressed well, as far as I can tell.

Unless it is clear by definition of "R" that "R" is *always* larger than LossFlightSize (more precisely, that the resulting cwnd is at least the RW=1 MSS), imho one has to deal with the case that the term cwnd = ((LossFlightSize - R)/2) can return a negative result or 0.

I don't fully understand what "retransmitted segments during the recovery phase for the first window of data" implies exactly, i.e., whether the cwnd could be negative or not.

> > implementing [Liu07] in Linux some years ago, I ran into situations 
> > where R was larger than LossFlightSize (possibly due to multiple 
> > retransmissions of the same segment). The equation cwnd = 
> > ((LossFlightSize - R)/2) could thus result in a negative cwnd. That 
> > issue might partly depend on how to exactly R is measured, 
> but I think 
> > that this equation requires further analysis and can't be 
> mandated by a MUST.
> >
> > Even if (LossFlightSize - R) is positive, I think that M. Allman's 
> > offlist comment about performance loss in loss of a short burst 
> > (Section 9.1) has to be addressed.
> >
> > For instance, I wonder why cwnd at the end of recovery 
> phase could not 
> > be reset to the value at the beginning, i.e.,
> >
> > If that value is a "safe" value for loss recovery, why is 
> it not save 
> > to use it after loss recovery?
> >
> - At the start of loss recovery, the sender needs to react at 
> least the same as normal TCP, by halving cwnd. But: cwnd/2 is 
> halving something that was not validated, and did not provide 
> any indication of the actual data sent - in the NVP, the cwnd 
> is only an upper limit.
> 
> -  pipeACK is intended as a measure of capacity used in the 
> recent past, but it can be quite different to (D-R) in cases 
> when the traffic pattern changes inducing congestion. It 
> helps though in not under-estimating capacity.
>
> - LossFlightSize (D) reflects the pattern that induced the 
> congestion - this is already in-flight during loss recovery.
> 
> The value of cwnd at the end of the loss recovery should not 
> be dependent on an initial cwnd that was not itself validated.

OK, this seems to make sense. But, still, why to calculate cwnd only based on LossFlightSize and not on pipeACK?

As far as I understand Mark's comment in Section 9.1, his concern is that pipeACK might be a more appropriate estimation than FlightSize in some cases. Maybe I miss something basic here, but this is not about "R" - it is about pipeACK vs. Flightsize.
 
For instance, something like this (I have not fully thought of it - maybe I am too naive here?):

  cwnd = Max(RW, Max(pipeACK-R, LossFlightSize-R)/2 )

Michael

> >> From: [email-masked] [mailto:[email-masked]] 
> On Behalf 
> >> Of [email-masked]
> congestion window 
> >> management algorithm has not changed since -00, but in -01 and -02 
> >> drafts we introduce a more robust way to measure the pipeACK that 
> >> attempts to account for the wide variety of interactive TCP 
> >> application behaviour.
> >> Clearly the cwnd poorly reflects the capacity in the non-validated 
> >> phase and I think it would be wrong to use this for TCP 
> control block 
> >> sharing.
> >> We therefore also  introduce text that proposes that TCB sharing 
> >> should use the pipeACK in place of cwnd when a TCP sender 
> is in the 
> >> Nonvalidated phase.  We think this value better reflects 
> the capacity 
> >> that the flow has utilised in the network path - but would welcome 
> >> more thought on this.
> >> > A New Internet-Draft is available from the on-line 
> Internet-Drafts 
> >> > directories.

---

From: Yuchung Cheng <[email-masked]>
Date: Mon, 29 Jul 2013 13:56:30 +0200

Hi Gorry,

I am interested in comparing this draft with this solution:

1. Do not reduce cwnd on idle (of any length)
2. From idle to active, send max(cwnd, data) over RTT in small bursts.
    E.g., cwnd=20, RTT= 100ms, send 4 packets every 20ms
    This emulates the ACK clocking.

TCP ack clocking offers smooth send. But it does not work well with
modern (structured) communication patterns. Often TCP does NOT have
data to send when it receives acks as the application uses persistent
connection or is thinking. This leads to bursts up to cwnd size.

If the idle interval is within an RTO, nothing can save the burst
including this draft. If it's longer than an RTO, this draft mitigates
it. But I would argue cutting cwnd is not addressing the fundamental
problem, because TCP still sends a cwnd of burst!

Burst induced losses are bad b/c it has not correlation to utilization
other than tricking TCP into congestion avoidance and low throughput.
Unfortunately it's hard to distinguish whether a loss is caused by
burst or congestion.

My sense is that any artificial idle length or cwnd/ssthresh cut
factor helps but does not scale well with time and bandwidth, but
maybe we can simply emulate ack clocking to really reduce the burst.
In some terms, it's either "always slow-start if ack clock is lost" or
"trust and send the old cwnd/rtt rate but not at line-rate burst". The
latter might be favored by Web transfers as the former is known to
kill Web performance.

Thoughts?

---

From: "Zimmermann, Alexander" <[email-masked]>
Date: Mon, 29 Jul 2013 12:44:25 +0000

Hi Yuchung,

>    This emulates the ACK clocking.

Smells like you propose a "IW=oo" here (with a straight forward burst
protection), isn't it?

> connection or is thinking. This leads to bursts up to cwnd size.

True, but this is a general problem, not only TCP CWV related.

> problem, because TCP still sends a cwnd of burst!

See above. Maybe it's worthwhile to tackle the burst problem in general.

---

From: Yuchung Cheng <[email-masked]>
Date: Mon, 29 Jul 2013 15:05:51 +0200

> Smells like you propose a "IW=oo" here (with a straight forward burst
No. I am proposing: when idle keep cwnd as is.

> True, but this is a general problem, not only TCP CWV related.

Yes. I am arguing that keep cwnd as-is is less a problem than the
burst generated w/o ack clocking in TCP. After all, there is a reason
cwnd grown to that so it's not a blind guess.

But when ack clock stops, you either trust the old rate (cwnd/rtt) or
conservatively always slow start with IW. What's the theory to do this
half way (e.g., reduce cwnd by X% after Y minute)?

Some anecdotal data: when we try the approach above (pace), the loss
rate is reduced and is as good as always slow-start after rto
(RFC5681), with better latency by keeping cwnd.

An expired draft has a great discussion on this.
http://www.isi.edu/touch/pubs/draft-hughes-restart-00.txt

---

From: "Zimmermann, Alexander" <[email-masked]>
Date: Mon, 29 Jul 2013 13:26:38 +0000

> No. I am proposing: when idle keep cwnd as is.

In a extreme case - let's say CWND = 100 and 7 days old - it's like IW=100
with burst protection.

The propose of CWND is to reflect the BDP. After 7 days the CWND is everything
but not the BDP.

---

From: Yuchung Cheng <[email-masked]>
Date: Mon, 29 Jul 2013 15:50:38 +0200

> The propose of CWND is to reflect the BDP. After 7 days the CWND is everything
In the not-so-extreme case, cutting cwnd to half after x minutes and
send 50 packets are not a good idea either.

I did say "OR slow start after IW". What I object is another cwnd
reduction after an artificial idle period. Slow start is fast bw probe
clocked by acks with an initial burst (=IW), I don't see a reason to
bump the burst size just b/c a sender has transmited something X
seconds ago, because it does not mean cwnd/N burst size works.

---

From: Hagen Paul Pfeifer <[email-masked]>
Date: Mon, 29 Jul 2013 16:31:00 +0200

>send 50 packets are not a good idea either.

The cwnd after idle could be a function of time - cwnd is already a function
of RTT (time) we could wide the mechanism for idle periods too.

Shortly after an active period the BDP is likely to be similar.
(slowstart/congestion avoidance is based on this assumption of time/BDP
correlation). After 10 minutes the BDP can vary significant as Alexander
points out. For some networks the assumption of a similar BDP is just wrong
(e.g TDMA based networks).

Hagen

---

From: Gorry Fairhurst <[email-masked]>
Date: Mon, 29 Jul 2013 16:39:32 +0200

This is a good question.

The "pacing" case is worth thinking through, but I'm not sure it is at 
all specific to cwv, see below.

I can see that a well-paced flow can in some cases experience lower 
loss. This is good for an app when there is a small amount of capacity 
available and good for the network also in this case. It does however 
reshape the app traffic, under all cases, even when there is no capacity 
limit - which suggests an incentive to build a large cwnd before you do 
this or to pad a flow to avoid this being triggered.

Pacing: First, if we pace bursts then we create a traffic shaper. That 
isn't much of an issue for a bulk sender, where the timing of individual 
packets is not important. If a resuming app runs for > RTT this 
recreates the ACK-Clock, but there again there are other Apps where this 
simply creates additional delay by attempting to create a well-paced 
stream. If the app is bursty in nature this impacts the latency - and 
there is the possibility that better performance can be got from the App 
perspective by padding the flow (not going silent), bursting before 
going idle (to inflate cwnd)  etc, to avoid the need to pace. This to me 
seems a bad incentive.

Pacing Interval: If we consider an app that can wait for some time to 
send, and therefore it is happy with it's bursts being smoothed, the 
case you chose cwnd=20, RTT = 100ms seems like it works out as a 
reasonable pacing interval. However a flow with a smaller cwnd and 
larger RTT would then get very shaped, and one with a smaller RTT e.g. 
10ms delay would imply a short pacing interval that would compete with 
times for transmit, checksum offloading, etc. I question whether your 
proposal is practical.

The "cwn-cut" after minutes is a mechanism to deal with long-term 
reduction of rate. It's not the main mechanism, but I think it is 
essential on longer timescales to avoid the problems of apps 
individually all building large cwnds and becoming idle for extended 
periods which then effectively means they have the ability to send 
whatever they like - for ever.

My fundamental question is do we need to reshape by pacing the app data 
bursts, to use TCP safely?

Gorry

---

From: Gorry Fairhurst <[email-masked]>
Date: Mon, 29 Jul 2013 18:07:39 +0200

> (e.g TDMA based networks).
Your proposal is similar to the RFC 2861 approach.
draft-ietf-tcpm-newcwv adopts a different approach.

Gorry

---

From: Andrew Knutsen <[email-masked]>
Date: Mon, 29 Jul 2013 12:34:56 -0700

I'd like to ask one question again, before we implement something to 
avoid issues with a customer.

    Most of the discussion here seems to be about avoiding bursts after 
idle while at the same time avoiding slow-start.

    The problem we have is not slow-start after idle; its going into 
congestion-avoidance after idle, due to an obsolete ssthresh.

    The current draft, like RFC2861, suggests setting ssthresh to 3/4 
the old cwnd value after idle.  We've tested this, and found that 
although it is an improvement over not doing anything (ie, rfc5681),  
cwnd does not recover with this factor when presented with a series of 
bursts if ssthresh is low.  We've also tested with setting it to  full 
cwnd, and to the advertised window (which is the receive window when 
idle).  Note that all of these are more conservative than closing and 
re-opening the connection, which sets ssthresh to infinity (if it isn't 
cached).

    We've found that setting cwnd to IW after idle, and setting ssthresh 
to awnd, allows acceptable recovery from a congestion event while 
running burst traffic.  In other words, slow-start works well enough 
from IW (it's fast compared to CA).

    We've tried to start discussion about this, but it looks like the 
consensus is a lack of concern.  While we would prefer to see a spec 
which addresses our concerns, we do have pressing need to fix this issue.

Andrew

---

From: Yuchung Cheng <[email-masked]>
Date: Mon, 29 Jul 2013 22:07:28 +0200

On Mon, Jul 29, 2013 at 9:34 PM, Andrew Knutsen
<[email-masked]> wrote:
>    Most of the discussion here seems to be about avoiding bursts after idle
> while at the same time avoiding slow-start.
>
>    The current draft, like RFC2861, suggests setting ssthresh to 3/4 the old
> cwnd value after idle.  We've tested this, and found that although it is an
> improvement over not doing anything (ie, rfc5681),  cwnd does not recover
> with this factor when presented with a series of bursts if ssthresh is low.
> We've also tested with setting it to  full cwnd, and to the advertised
> window (which is the receive window when idle).  Note that all of these are
> more conservative than closing and re-opening the connection, which sets
> ssthresh to infinity (if it isn't cached).
>
>    We've found that setting cwnd to IW after idle, and setting ssthresh to
> awnd, allows acceptable recovery from a congestion event while running burst
> traffic.  In other words, slow-start works well enough from IW (it's fast
> compared to CA).

Ah I understand your problem now. Since RFC5681 section 4.1 does not
specify what to do with ssthresh. Why not set to infinity?

I would like to point out that fundamentally initial cwnd (IW) == idle
cwnd (RW) because ack clock is stopped. If RW can be > IW, I argue IW
should be further increased.

But there is the idea of cwnd (or rate) caching which enhances
performance. So I think the middle-ground is retain cwnd but pace over
an RTT.

"" 4.1.  Restarting Idle Connections

   A known problem with the TCP congestion control algorithms described
   above is that they allow a potentially inappropriate burst of traffic thi
   to be transmitted after TCP has been idle for a relatively long
   period of time.  After an idle period, TCP cannot use the ACK clock
   to strobe new segments into the network, as all the ACKs have drained
   from the network.  Therefore, as specified above, TCP can potentially
   send a cwnd-size line-rate burst into the network after an idle
   period.  In addition, changing network conditions may have rendered
   TCP's notion of the available end-to-end network capacity between two
   endpoints, as estimated by cwnd, inaccurate during the course of a
   long idle period.
""

> consensus is a lack of concern.  While we would prefer to see a spec which
> addresses our concerns, we do have pressing need to fix this issue.
>
> Andrew
>

---

From: Yuchung Cheng <[email-masked]>
Date: Mon, 29 Jul 2013 22:25:16 +0200

On Mon, Jul 29, 2013 at 4:39 PM, Gorry Fairhurst <[email-masked]> wrote:
> This is a good question.
>
> The "pacing" case is worth thinking through, but I'm not sure it is at all
> specific to cwv, see below.
>
> I can see that a well-paced flow can in some cases experience lower loss.
> This is good for an app when there is a small amount of capacity available
> and good for the network also in this case. It does however reshape the app
> traffic, under all cases, even when there is no capacity limit - which
> suggests an incentive to build a large cwnd before you do this or to pad a
> flow to avoid this being triggered.
>
> Pacing: First, if we pace bursts then we create a traffic shaper. That isn't
> much of an issue for a bulk sender, where the timing of individual packets
> is not important. If a resuming app runs for > RTT this recreates the
> ACK-Clock, but there again there are other Apps where this simply creates
> additional delay by attempting to create a well-paced stream. If the app is
> bursty in nature this impacts the latency - and there is the possibility
> that better performance can be got from the App perspective by padding the
> flow (not going silent), bursting before going idle (to inflate cwnd)  etc,
> to avoid the need to pace. This to me seems a bad incentive.
See my reply to Andrew's question. The original reason of RW in
RFC5681 is to avoid burst.

>
> Pacing Interval: If we consider an app that can wait for some time to send,
> and therefore it is happy with it's bursts being smoothed, the case you
> chose cwnd=20, RTT = 100ms seems like it works out as a reasonable pacing
> interval. However a flow with a smaller cwnd and larger RTT would then get
> very shaped, and one with a smaller RTT e.g. 10ms delay would imply a short
> pacing interval that would compete with times for transmit, checksum
> offloading, etc. I question whether your proposal is practical.
Only running code can answer your question :)

>
> The "cwn-cut" after minutes is a mechanism to deal with long-term reduction
> of rate. It's not the main mechanism, but I think it is essential on longer
> timescales to avoid the problems of apps individually all building large
> cwnds and becoming idle for extended periods which then effectively means
> they have the ability to send whatever they like - for ever.
>

We don't need to. The Internet is running w/o this today. It's just (a
lot of) line-rate burst induced losses :(

>>>>>>> drafts
>>>>>>> phase
>>>>>>> should

---

From: Andrew Knutsen <[email-masked]>
Date: Mon, 29 Jul 2013 13:36:44 -0700

It is possible that there are ways to retain cwnd after idle and use 
some other mechanism to avoid inappropriate bursts. However, in the 
cases we've seen, slow-start from IW works just fine for this.  Its CA 
from IW we have issues with, and it seems like there are simple solutions.

    I don't mean to say there isn't room for more complex solutions 
involving ramping cwnd and perhaps simulating ack clocking, but its 
seems like we'll still want to deal with ssthresh.

Andrew

---

From: Anna Brunstrom <[email-masked]>
Date: Tue, 30 Jul 2013 00:35:02 +0200

Hi Andrew,

I agree with you that the value of ssthresh is important and that this 
part may need to be updated. If subsequent bursts are not long enough to 
take you out of an unlucky state, a random loss in the the first burst 
may hamper performance for all later bursts by invoking CA too early.

We have seen the equivalent problem when you have a sequence of short 
flows and the ssthresh value is cached. A random loss in one short flow 
can reduce performance for all the short flows that follow. See our ICC 
paper from last year for details 
http://dx.doi.org/10.1109/ICC.2012.6364516.

BR,
Anna

---

From: Joe Touch <[email-masked]>
Date: Mon, 29 Jul 2013 17:35:54 -0700

> http://www.isi.edu/touch/pubs/draft-hughes-restart-00.txt

Yes, but it proposes solutions that are not being considered, AFAICT.

That draft was concerned *anytime* the window grew beyond what was being 
actively used, regardless of whether that happens all at once (idle 
period) or over time (e.g., a source that can't send as much as CWND 
allows, but the ACKs keep accumulating permission to burst).

We felt that the better solution was to limit the ability to accumulate 
that permission. Note that this was also an issue with Lixia's original 
virtual clock mechanism, FWIW.

I'm still not sure why this isn't relevant here; is it more useful to 
validate a large unused permission-to-send window, or just prohibit 
hoarding that as a possible resource?

Joe

---

From: Joe Touch <[email-masked]>
Date: Mon, 29 Jul 2013 17:38:20 -0700

> all specific to cwv, see below.

In this doc, http://www.isi.edu/touch/pubs/draft-hughes-restart-00.txt, 
we considered pacing too, but it was suboptimal for a number of reasons 
(complexity, the need to determine additional parameters, and the lag in 
response time).

Joe

---

From: Yuchung Cheng <[email-masked]>
Date: Tue, 30 Jul 2013 07:21:18 +0200

On Mon, Jul 29, 2013 at 10:36 PM, Andrew Knutsen
<[email-masked]> wrote:
>
>    It is possible that there are ways to retain cwnd after idle and use some
> other mechanism to avoid inappropriate bursts. However, in the cases we've
> seen, slow-start from IW works just fine for this.  Its CA from IW we have
> issues with, and it seems like there are simple solutions.
+1

>
>    I don't mean to say there isn't room for more complex solutions involving
> ramping cwnd and perhaps simulating ack clocking, but its seems like we'll
> still want to deal with ssthresh.
yes. it will be nice get some empirical study of setting ssthresh to
inifinity after idle

>>> idle
>>> to
>>> which

---

From: <[email-masked]>
Date: Tue, 30 Jul 2013 09:38:11 +0300

HI,

We also have observed issues with ssthresh setting after idle.
This regards both TCP and SCTP usecase scenarios.

Our thoughts have been to reset ssthresh to inifinity after idle.
Setting to awnd is not a good solution in SCTP (or for mptcp) as the awnd 
may be small even if an idle period has been observed on a specific path on which traffic now resumes.

Possible SCTP or multipath issues are not within cwv, but still I would like to point to this fact.

BR, Karen

>From: [email-masked] [mailto:[email-masked]] On Behalf Of Anna
>Brunstrom
>Sent: Tuesday, July 30, 2013 12:35 AM
>To: [email-masked]
>Subject: Re: [tcpm] New revision: draft-ietf-tcpm-newcwv-02.txt: ssthresh
>
>Hi Andrew,
>
>I agree with you that the value of ssthresh is important and that this part may need
>to be updated. If subsequent bursts are not long enough to take you out of an
>unlucky state, a random loss in the the first burst may hamper performance for all
>later bursts by invoking CA too early.
>
>We have seen the equivalent problem when you have a sequence of short flows and
>the ssthresh value is cached. A random loss in one short flow can reduce
>performance for all the short flows that follow. See our ICC paper from last year for
>details http://dx.doi.org/10.1109/ICC.2012.6364516.
>
>BR,
>Anna
>
>

---

From: Michael Welzl <[email-masked]>
Date: Tue, 30 Jul 2013 10:12:35 +0200

Hi,

In the discussion in TCPM just now, it seemed that people look at a connection that goes idle and later sends again as being roughly the same thing as a connection that stops and a new connection that starts. But: if there is a box in the path that does ECMP, this box may use the SYN and/or the use of a new port number (if only on the client side) as a basis to pick a path, and so a second connection starting after one that stopped may traverse a different path, right?

If I'm right then this is perhaps quite a different thing…  (but note that I have no experience with boxes doing stuff like I'm writing here, this is just guesswork)

Cheers,
Michael

---

From: Andrew Mcgregor <[email-masked]>
Date: Tue, 30 Jul 2013 01:18:22 -0700

You're correct in that changing the port number on one end will probably
cause a different ECMP path to become active.  However, in most cases the
operator is trying very hard to keep the ECMP paths balanced, so I believe
this is OK.  If not, they're doing complex traffic engineering, and chances
are the path can change mid-connection anyway as the TE process reacts to
load.  So I don't believe this to be a serious issue in either case.

On 30 July 2013 01:12, Michael Welzl <[email-masked]> wrote:

> Hi,
>
> In the discussion in TCPM just now, it seemed that people look at a
> connection that goes idle and later sends again as being roughly the same
> thing as a connection that stops and a new connection that starts. But: if
> there is a box in the path that does ECMP, this box may use the SYN and/or
> the use of a new port number (if only on the client side) as a basis to
> pick a path, and so a second connection starting after one that stopped may
> traverse a different path, right?
>
> If I'm right then this is perhaps quite a different thing…  (but note that
> I have no experience with boxes doing stuff like I'm writing here, this is
> just guesswork)
>
> Cheers,
> Michael
>

-- 
Andrew McGregor | SRE | [email-masked] | +61 4 8143 7128

---

From: Gorry Fairhurst <[email-masked]>
Date: Tue, 30 Jul 2013 10:30:33 +0200

I agree. In theory a new port generates new entropy that *could* result 
in a new ECMP path. I'm not sure this is a new major issue for cwv.

Gorry

On 30/07/2013 10:18, Andrew Mcgregor wrote:
> cause a different ECMP path to become active.  However, in most cases
> the operator is trying very hard to keep the ECMP paths balanced, so I
> believe this is OK.  If not, they're doing complex traffic engineering,
> and chances are the path can change mid-connection anyway as the TE
> process reacts to load.  So I don't believe this to be a serious issue
> in either case.
>
>
> On 30 July 2013 01:12, Michael Welzl <[email-masked]
> <mailto:[email-masked]>> wrote:
>
>     Hi,
>
>     connection that goes idle and later sends again as being roughly the
>     same thing as a connection that stops and a new connection that
>     starts. But: if there is a box in the path that does ECMP, this box
>     may use the SYN and/or the use of a new port number (if only on the
>     client side) as a basis to pick a path, and so a second connection
>     starting after one that stopped may traverse a different path, right?
>
>     If I'm right then this is perhaps quite a different thing…  (but
>     note that I have no experience with boxes doing stuff like I'm
>     writing here, this is just guesswork)
>
>     Cheers,
>     Michael
>
>     [email-masked] <mailto:[email-masked]>
> Andrew McGregor | SRE [email-masked]
> <mailto:[email-masked]> | +61 4 8143 7128
>
>

---

From: Michael Welzl <[email-masked]>
Date: Tue, 30 Jul 2013 11:08:14 +0200

It isn't an issue for cwv, but it may be a reason why the logic: "if we allow such a large burst for cwv, we could just as well allow an even larger IW" may not hold.

Cheers,
Michael

On Jul 30, 2013, at 10:30 AM, Gorry Fairhurst <[email-masked]> wrote:

> I agree. In theory a new port generates new entropy that *could* result in a new ECMP path. I'm not sure this is a new major issue for cwv.
> 
> Gorry
>

---

From: Joe Touch <[email-masked]>
Date: Tue, 30 Jul 2013 10:27:54 -0700

On 7/30/2013 1:12 AM, Michael Welzl wrote:
> same thing as a connection that stops and a new connection that starts.

That was the assumption when the current code was designed, but we know 
better now. We can optimize across successive connections (TCB control 
block sharing), and we can do better during even brief periods of idle.

Note that the burst after idle is a specific case of a more general 
condition that doesn't require true idle to occur - it just requires a 
source that doesn't use the accumulated "permission to send" implied by 
an increasing congestion window.

Joe

---

From: Andrew Knutsen <[email-masked]>
Date: Tue, 30 Jul 2013 18:03:15 -0700

Given this observation, and the apparent general agreement that 
reducing bursts while avoiding slow-start is more important, I'm going 
to go ahead and bump ssthresh higher than recommended by the draft (and 
RFC2861) in our TCP.  Increasing the RW can wait for the draft to be 
finalized, if we need it.

    Thanks for your attention.

Andrew

---

From: "Scharf, Michael (Michael)" <[email-masked]>
Date: Wed, 31 Jul 2013 07:01:23 +0000

Hi Andrew,

I wonder if you have experimented with setting ssthresh after an idle period to a value that is of the order of the pipeACK parameter (cf. draft-ietf-tcpm-newcwv-02 Section 4.2 - not flightsize)?

(Yes, I am aware of several other alternatives.)

Michael (without blue dot)

> Behalf Of Andrew Knutsen
> Sent: Wednesday, July 31, 2013 3:03 AM
> To: [email-masked]
>     Given this observation, and the apparent general 
> agreement that reducing bursts while avoiding slow-start is 
> more important, I'm going to go ahead and bump ssthresh 
> higher than recommended by the draft (and
> RFC2861) in our TCP.  Increasing the RW can wait for the 
> draft to be finalized, if we need it.
> 
> > Setting to awnd is not a good solution in SCTP (or for 
> mptcp) as the 
> > awnd may be small even if an idle period has been observed 
> on a specific path on which traffic now resumes.
> >
> > Possible SCTP or multipath issues are not within cwv, but 
> still I would like to point to this fact.
> >
> > BR, Karen
> >
> >
> >> Of Anna Brunstrom
> >> I agree with you that the value of ssthresh is important and that 
> >> this part may need to be updated. If subsequent bursts are 
> not long 
> >> enough to take you out of an unlucky state, a random loss 
> in the the 
> >> first burst may hamper performance for all later bursts by 
> invoking CA too early.
> >>
> >> flows and the ssthresh value is cached. A random loss in one short 
> >> flow can reduce performance for all the short flows that 
> follow. See 
> >> our ICC paper from last year for details 
> >>>      I'd like to ask one question again, before we implement 
> >>> something to avoid issues with a customer.
> >>>
> >>>      Most of the discussion here seems to be about 
> avoiding bursts 
> >>> after idle while at the same time avoiding slow-start.
> >>>
> >>>      The problem we have is not slow-start after idle; its going 
> >>> into congestion-avoidance after idle, due to an obsolete ssthresh.
> >>>
> >>>      The current draft, like RFC2861, suggests setting 
> ssthresh to 
> >>> 3/4 the old cwnd value after idle.  We've tested this, and found 
> >>> that although it is an improvement over not doing anything (ie, 
> >>> rfc5681), cwnd does not recover with this factor when 
> presented with 
> >>> a series of bursts if ssthresh is low.  We've also tested with 
> >>> setting it to  full cwnd, and to the advertised window 
> (which is the 
> >>> receive window when idle).  Note that all of these are more 
> >>> conservative than closing and re-opening the connection, 
> which sets 
> >>>      We've found that setting cwnd to IW after idle, and setting 
> >>> ssthresh to awnd, allows acceptable recovery from a 
> congestion event 
> >>> while running burst traffic.  In other words, slow-start 
> works well 
> >>> enough from IW (it's fast compared to CA).
> >>>
> >>>      We've tried to start discussion about this, but it 
> looks like 
> >>> the consensus is a lack of concern.  While we would 
> prefer to see a 
> >>> spec which addresses our concerns, we do have pressing 
> need to fix this issue.
> >>>
> >>> Andrew
> >>>

---

From: Gorry <[email-masked]>
Date: Wed, 31 Jul 2013 20:35:40 +0200

So,

Here is my own current "idea" ....

we think that cwv needs one mechanism to reduce cwnd and at the same time increase ssthresh to 3/4 of the value prior to reduction. This is a separate problem.

There is another case when there is a long time since the last congestion event (define long) and we should then allow ssthresh to be reset to its initial ssthresh. As you suggest, important for many use cases.

Another case, an idle app - which could also do an initial ssthresh reset. This is a specific case of above, but one that is much easier to detect (for an app that goes idle).

There is possibly a Small cwnd case, if the cwnd if less than a few IW, then maybe there is a perhaps a case for a reset of ssthresh after a shorter time.

That is a lot of different options... We have many inputs and we think we can propose text on the list for new-cwv, and then see if this is ok with people.

Gorry