Subject: [tcpm] TLP

From: "Scheffenegger, Richard" <[email-masked]>
Date: Fri, 15 Mar 2013 16:41:49 +0000

Hi Nandita, Yuchung, Neal,

this might be a stupid idea, but I bring it up anyway...

TLP is to elicit an ACK from the receiver at the end of a burst, where some of the ultimate segments might have been lost...

What would happen, if the sender (which knows the size of the burst, and if the stream actually ends...) would deliberately send the 3 final packets of the burst in reverse order,

Instead of ... 6 7 8 9, send 6, 9, 8, 7, or 9, 6, 8, 7 (interleaving avoids exceeding/modifying dupthresh in the no-loss case, and can be extended to the last N segments).

The slides your colleague showed at some point the probability of a packet loss into a burst, which linearly rises into the length of the ....

As one critical issue is to have the receiver "know" how many packets should come, and to remove the receiver side delACK timer, a deliberate reordering might achieve this without using timers and blind sending of one (two) packets to the receiver...

(Implementing this with HW support is probably more challenging though).

Any thoughts on that? Was that route investigated, and if so, what drawbacks did you identify?

Best regards,

Richard Scheffenegger

NetApp
[email-masked]<mailto:[email-masked]>
+43 1 3676811 3146 Office (2143 3146 - internal)
+43 676 654 3146 Mobile
www.netapp.com<http://www.netapp.com>

EURO PLAZA
Gebäude G, Stiege 7, 3.OG
Am Euro Platz 2
A-1120 Wien

---

From: Anthony Sabatini <[email-masked]>
Date: Fri, 15 Mar 2013 13:12:39 -0400

All -

I agree with RS, an important point here is that in a dual network link or alternate route scenario you often get just this out of order sequence of messages.  This is why I counseled the use of a "settling time", a period for all relevant packets to arrive in before making a decision so that the ordering issues would not affect link retransmission calculations (specifically by attaching a timestamp to each incoming packet and not acting on the implication of that packet until a set amount of time later).

Tony

Anthony Sabatini
200 West 20th Street
Apt. 1216
New York, NY 10011

Phone: (212) 867-7179
Mobile: (917) 224-8388

From: [email-masked]
To: [email-masked]; [email-masked]; [email-masked]
Date: Fri, 15 Mar 2013 16:41:49 +0000
CC: [email-masked]
Subject: [tcpm] TLP

 
TLP is to elicit an ACK from the receiver at the end of a burst, where some of the ultimate segments might have been lost…
 
What would happen, if the sender (which knows the size of the burst, and if the stream actually ends…) would deliberately send the 3 final packets of the burst in reverse order,
 
Instead of … 6 7 8 9, send 6, 9, 8, 7, or 9, 6, 8, 7 (interleaving avoids exceeding/modifying dupthresh in the no-loss case, and can be extended to the last N segments).
 
The slides your colleague showed at some point the probability of a packet loss into a burst, which linearly rises into the length of the ….
 
As one critical issue is to have the receiver “know” how many packets should come, and to remove the receiver side delACK timer, a deliberate reordering might achieve this without using timers and blind sending of one (two) packets to the receiver…

 
Best regards,
 
Richard Scheffenegger

NetApp

[email-masked]

+43 676 654 3146 Mobile

www.netapp.com  

EURO PLAZA

Am Euro Platz 2

A-1120 Wien

_______________________________________________
tcpm mailing list
[email-masked]
https://www.ietf.org/mailman/listinfo/tcpm

---

From: Jakob Heitz <[email-masked]>
Date: Fri, 15 Mar 2013 17:38:48 +0000

Nice!

--
Jakob Heitz.

________________________________
From: [email-masked] [mailto:[email-masked]] On Behalf Of Scheffenegger, Richard
Sent: Friday, March 15, 2013 9:42 AM
To: Yuchung Cheng; Nandita Dukkipati ([email-masked]); Neal Cardwell ([email-masked])
Cc: tcpm ([email-masked])
Subject: [tcpm] TLP

Best regards,

Richard Scheffenegger

NetApp

EURO PLAZA

---

From: Yuchung Cheng <[email-masked]>
Date: Fri, 15 Mar 2013 11:05:16 -0700

On Fri, Mar 15, 2013 at 9:41 AM, Scheffenegger, Richard <[email-masked]> wrote:
> TLP is to elicit an ACK from the receiver at the end of a burst, where some
> of the ultimate segments might have been lost…
>
> What would happen, if the sender (which knows the size of the burst, and if
> the stream actually ends…) would deliberately send the 3 final packets of
> the burst in reverse order,
>
> Instead of … 6 7 8 9, send 6, 9, 8, 7, or 9, 6, 8, 7 (interleaving avoids
> exceeding/modifying dupthresh in the no-loss case, and can be extended to
> the last N segments).
I am not sure how this addresses the tail drops. If the sender sends 9,6,8,7
then 7 got lost, not only the sender needs to be careful not triggering fast
recovery on receiving DUPACK of SACK9 or SACK8, the sender ultimately
has to wait and retry 7. Similar case if 8,7 got lost.

TLP is to quickly resume the sending rate on small tail drops if the
ack clocking
is not completely lost (within 1-2 RTTs).  Tail drop is about packets
not sequences.
When the last packets, regardless of their TCP sequences, are dropped the
sender needs to wait a reasonable time and retry. How much time is reasonable
is worth discussing, but playing with sequence numbers don't help.

The probe packet does not have to be the last packet. The reason we choose
that is convenience to trigger FACK-sender's Fast Recovery w/o any changes.

Also, without any losses, receiving 9,6,8,7 drives receiver into
slow-path processing.

>
> The slides your colleague showed at some point the probability of a packet
> loss into a burst, which linearly rises into the length of the ….
>
> As one critical issue is to have the receiver “know” how many packets should
> come, and to remove the receiver side delACK timer, a deliberate reordering
> might achieve this without using timers and blind sending of one (two)
> packets to the receiver…
>

---

From: "Scheffenegger, Richard" <[email-masked]>
Date: Fri, 15 Mar 2013 22:43:16 +0000

Hi Yuchung,

True,

but you don't need to wait for a (unknown) delAck timeout of the receiver in the special case. 

Also, the last few ACKs will be received by the sender ideally with similar time delay between them, so assuming the path didn't change you could do a slightly faster PTO. Further, by the time of the PTO you already know how many packets to retransmit (if one, two or more packets didn't made it) because the scoreboard has been populated already. With TLP, only the TLP packet populates the scoreboard (thus that ACK is more critical).

I assumed TCP SACK (just as in TLP) for this to work, not NewReno...

Yes, both sender and receiver would be processing these last packets of a burst in slow-path (which might not be the worst idea, actually, to slightly pace out on the sender side; again, if the cumulative ACK doesn't show by the PTO time, you then know already (usually) which packets to retransmit. So with 2 or more packet drops, you save one RTT recovering them, not?

Only the case with the very last packet being lost, the regular and TLP recovery time would be the same.

Of course, I haven't looked in detail in all the scenarios, but the macroscopic behavior might be beneficial for your particular use case (saving RTT on multiple losses), and the microscopic behavior beneficial to the network (less spurious packets)?

Best regards,

Richard Scheffenegger

> -----Original Message-----
> From: Yuchung Cheng [mailto:[email-masked]]
> Sent: Freitag, 15. März 2013 19:05
> To: Scheffenegger, Richard
> Cc: Nandita Dukkipati ([email-masked]); Neal Cardwell
> ([email-masked]); tcpm ([email-masked])
> Subject: Re: TLP
> 
> > TLP is to elicit an ACK from the receiver at the end of a burst, where
> > some of the ultimate segments might have been lost.
> >
> > What would happen, if the sender (which knows the size of the burst,
> > and if the stream actually ends.) would deliberately send the 3 final
> > packets of the burst in reverse order,
> >
> > Instead of . 6 7 8 9, send 6, 9, 8, 7, or 9, 6, 8, 7 (interleaving
> > avoids exceeding/modifying dupthresh in the no-loss case, and can be
> > extended to the last N segments).
> I am not sure how this addresses the tail drops. If the sender sends
> 9,6,8,7 then 7 got lost, not only the sender needs to be careful not
> triggering fast recovery on receiving DUPACK of SACK9 or SACK8, the sender
> ultimately has to wait and retry 7. Similar case if 8,7 got lost.
> 
> TLP is to quickly resume the sending rate on small tail drops if the ack
> clocking is not completely lost (within 1-2 RTTs).  Tail drop is about
> packets not sequences.
> sender needs to wait a reasonable time and retry. How much time is
> reasonable is worth discussing, but playing with sequence numbers don't
> help.
> 
> Also, without any losses, receiving 9,6,8,7 drives receiver into slow-path
> processing.
> 
> >
> > The slides your colleague showed at some point the probability of a
> > packet loss into a burst, which linearly rises into the length of the ..
> >
> > As one critical issue is to have the receiver "know" how many packets
> > should come, and to remove the receiver side delACK timer, a
> > deliberate reordering might achieve this without using timers and
> > blind sending of one (two) packets to the receiver.
> >

---

From: "Scheffenegger, Richard" <[email-masked]>
Date: Mon, 18 Mar 2013 11:24:41 +0000

Hi Yuchung,

Here is a short comparison between the TLP draft behavior and the deliberate tail reordering I mentioned.

A PTO timer will still be necessary, but again, one special case can be avoided (unless dealing with very thin streams, which are app limited to <=1 segment/RTT), and full recovery is accomplished at least 1 RTT sooner than with TLP, provided that cwnd is large enough. I hope the ascii art survives (use fixed-font); the PTO/delack delay are not to scale with RTT though.

     draft TLP                      deliberate reordering              
                                    TLP                                
                                                                       
     5-_                            5-_                                
        ^-_                            ^-_                             
     6-_   ^-_                      9-_   ^-_                          
        ^-_   ^-_                      ^-_   ^-_                       
     7-_   ^-_   ^->                6-_   ^-_   ^->  (delayed ACK)                  
        ^-_   ^-_                      ^-_   ^-_                       
     8-X   ^-_   ^->                8-X   ^-_   ^->                    
              ^-_  _-6                       ^-_  _-5(9)               
     9-X        _^^>   -+           7-X        _^^>                    
             _-^        |                   _-^   _-6(9)               
          _-^           delAck TO        _-^   _-^                     
       <-^              |             <-^   _-^                        
                   _-7 -+                _-^                           
                _-^              +-   <-^                              
             _-^                 |                                     
          _-^                    P                                     
  +-   <-^                       T                                     
  |                              O                                     
  P                              |                                     
  T                              +- 7-_                                
  O                                    ^-_                             
  |                                 8-_   ^-_                          
  +- 9-_                               ^-_   ^-_                       
        ^-_                               ^-_   ^->                    
           ^-_                               ^-_  _-7(9)               
              ^-_                               ^->                    
                 ^->                              _-9                  
                   _-7(9)                                              
                _-^                                                    
             _-^                                                       
          _-^                                                          
       <-^                                                             
     8-_                                                               
        ^-_                                                            
           ^-_                                                         
              ^-_                                                      
                 ^->                                                   
                    -9                                                 

Also, the graph showing relative loss probability into a tail end burst indicates to me, that the later packets of a burst are more likely to really have been dropped... Thus the retransmission sequence might be modified to re-send the last packets first (TLP resending the ultimate packet only performs this).

Also note, that with the deliberate reordering, the sender already knows at the time when PTO fires (after being rescheduled with the arriving ACKs), which segments are most likely still outstanding. If cwnd permits, more than a single segment may be re-sent when PTO fires.

The reordering also causes the sender to send more ACKs (inter-ACK arrival times at the sender could be used as additional signal). If the cumulative (last) ACK is lost, the special handling to detect if a PTO retransmission was necessary, is still necessary. Again, if the PTO retransmission was unnecessary, and TS is used, an old TS (since the retransmission doesn't advance the receiver window), or DSACK block would be strong signals...

Best regards,

Richard Scheffenegger

---

From: "Scheffenegger, Richard" <[email-masked]>
Date: Wed, 20 Mar 2013 13:35:57 +0000

Definitely...

But the use case mentioned by the authors of TLP is the timely delivery of web objects. So, the sender does know the probably end of the burst (the sender queue is not fully utilized) and could perform that deliberate reordering.

In the degenerate case, that the application "tickles in" data in small chunks that fit into a single packet (or Nigle is turned off), the application probably does some pacing already, so that the tail-loss-after-burst, that is to be addressed, is most likely less of an issue.

In a "burst" size of one packet, there is nothing to reorder, but the chance that this segment is lost, is also reduced (I cann't find the presentation, where the probability of subsequent packet losses, after the first packet loss, was given vs. the burst size).

In that case, TLP would send the ultimate segment anew, if no cumulative ACK was seen during the PTO period...

But with deliberate reordering of the ultimate (and penultimate) segments, the deliberate reordering should yield the exact information (except of ACK losses, of course), which segments to retransmit, and also would deliver a "better" ACK clock by disabling delayed ACKs...

I hope that point came across in my last mail with the ascii tcp diagrams...

Best regards,

Richard Scheffenegger

NetApp
[email-masked]

EURO PLAZA

> From: Rick Jones [mailto:[email-masked]]
> Sent: Montag, 18. März 2013 21:38
> Subject: Re: [tcpm] TLP
> I missed the beginning of the discussion, so I'll ask this question
> directly lest it be an Emily Litella Moment - doesn't deliberate
> reordering rely on having all the data in the first place rather than
> having it dribbling-in?
> 
> rick jones