Subject: Re: [tcpm] Vicious or Virtuous circle? Adapting reordering window to reordering degree

From: Bob Briscoe <[email-masked]>
Date: Fri, 6 Apr 2018 21:55:48 +0100

Michael,

Returning to this after a week tuned out (for which apologies)...

On 29/03/18 12:42, Michael Welzl wrote:
> Hi all,
>
> I was just about to propose a socket option for the upper limit on 
> reordering (e.g. as a function of the RTT) - to convey whether an 
> application is latency-critical or not. However, some of what Bob 
> wrote below throws me off… I keep only the relevant bits and 
> reordering them slightly for better context:
>
>
>> On Mar 29, 2018, at 11:05 AM, Bob Briscoe <[email-masked] 
>> <mailto:[email-masked]>> wrote:
>>
>> Matt,
>>
>>  1. bandwidth inefficiency depends on spurious retransmissions, which
>>     *reduce* as reordering tolerance increases
>>  2. loss recovery delay *increases* as reordering tolerance increases
>>     {Note 1}
>>  3. delivery delay *reduces* as reordering tolerance increases {Note 2}
>>
> [snip]
>>
>>
>> {Note 1}: There are other ways to remove losses, e.g. FEC, ECN. In an 
>> ideal world, if they were 100% successfully employed, there would be 
>> no trade off - increasing reordering tolerance would be wholly 
>> beneficial.
>
> The second sentence here confuses me: undoubtedly, for an application 
> that needs to get stuff in order, reordering can cause delay 
> (considering only one connection). That is, if it is produced e.g. by 
> in-network load balancing that sends some data over a longer-latency 
> path, that delay affects everything. E.g. ECN can’t help that?!
[BB] I think you've overlooked the word "tolerance" in my Note 1.

I was using the terminology Yuchung has been using:
* Reordering degree = the amount of reordering introduced by the network
* Reordering tolerance = the amount of reordering tolerated by the 
sender before retransmitting.

If the sender increases its reordering tolerance, it waits longer before 
it retransmits, which increases the delay before losses get repaired.

My Note 1 merely pointed out that, in an ideal world with ECN and FEC, 
retransmission would not be needed to repair losses, so reordering 
tolerance could be arbitrarily high. No need to sweat this point - it's 
theoretical, which is why I relegated it to a footnote (double irony: 
your reordering of my footnotes did not improve comprehension).

>
>
>> {Note 2}: Shifting reordering to the end-system reduces head-of-line 
>> blocking of one stream when another has suffered a loss because it's 
>> the receiver's job to demultiplex between microflows, which naturally 
>> removes intra-flow HoL blocking. Also, if there are streams within an 
>> encrypted microflow (e.g. http2, quic), only the receiver can 
>> demultiplex the streams to resolve any HoL blocking.
>
> But we’re talking about TCP here, and generally I find this paragraph 
> hard to parse. You seem to have a multi-flow scenario in mind that 
> isn’t fully described. To begin with, without Minion’ish trickery, a 
> single TCP connection can only deliver data that arrives in order up 
> to the application, and hence reordering (when some data arrive later, 
> not earlier, because they use a longer-latency path) can cause delay 
> for that application.
[BB] Actually, there's a bigger problem with my 3rd point - it's at a 
completely different timescale to the other two.
* My first 2 trade offs occur immediately, at run-time, as soon as a 
particular sender alters its reordering tolerance.
* Whereas the 3rd trade off occurs years after senders generally have 
increased their reordering tolerance (by deploying RACK) - once networks 
have evolved in response.
Sorry I didn't make that clear.

Actually, the first sentence of Note 2 was meant to apply to any 
transport protocol including TCP. Except I meant to write inter-flow, 
not intra-flow, so I don't blame you for being confused.

You're right though about the second sentence - it does not apply to 
TCP, because it's about resolving HoL blocking between intra-flow 
streams, which TCP can't do.

Sorry about both screw-ups. I tried to write that email quite precisely 
and carefully :(

>
>
> A separate thing:
>
>> However, as I said in a previous posting, this doesn't address the 
>> *cost* trade off in processing and memory terms:
>>
>>  1. increasing reordering tolerance increases *receiver* memory &
>>     processing costs
>>  2. reducing reordering tolerance increases *network* memory and
>>     processing costs (consequently capping network throughput)
>>
> About 1, isn’t this perhaps solved by your rwnd idea (your email to 
> Praveen)?
[BB] Well, the rwnd idea allows the receiver to control its cost, 
otherwise control of reordering tolerance is purely with the sender.

But this doesn't "solve" the bigger problem. It shifts the cost to the 
network (by making the sender reduce its reordering tolerance, which 
shifts to #2 scenario). But it doesn't make the host/network trade off 
go away.

> If it is, then the answer to 2 is: fine, we can increase reordering 
> tolerance anyway.
>
> Generally, I agree with the pro-reordering musings in this thread, 
> e.g. from Matt - we have a silly situation in the network today, where 
> TCP acts as if it was a connection oriented network and in-network 
> devices jump through various hoops to maintain this illusion. In fact, 
> the Internet is a packet oriented network, and if we can tweak things 
> a little in a direction that allows people to tune the network in a 
> more packet oriented fashion (instead of TCP-connection-oriented), 
> then that’s a good thing.
>
> Looking at the draft, RACK asks for Experimental status. Couldn’t we 
> see "finding out how much reordering is acceptable” as part of the 
> experiment?  I mean, some version of RACK is in Linux already, and 
> things haven’t broken badly as far as I know.  This experiment would 
> at least let us uncover how severe the processing / memory costs above 
> really are.
[BB] ...true, with the proviso that, over any reasonable experiment 
timescale, you wouldn't see the network evolve much, so you wouldn't 
know how much greater reordering might get once the rules for in-order 
forwarding are relaxed.

>
> Cheers,
> ichMael   (Michael, reordered)
>
[BB] Cheers

boB

-- 
________________________________________________________________
Bob Briscoe                               http://bobbriscoe.net/

---

From: Joe Touch <[email-masked]>
Date: Sun, 8 Apr 2018 12:55:58 -0700

> On Apr 6, 2018, at 1:55 PM, Bob Briscoe <[email-masked]> wrote:
> 
> My Note 1 merely pointed out that, in an ideal world with ECN and FEC, retransmission would not be needed to repair losses,

ECN + FEC is not a guarantee of E2E transmission. Packets are lost for reasons other than congestion, and no amount of FEC overcomes losses that exceed what that FEC anticipates.

Joe