Subject: Re: [tcpm] Inner Option Space revised: inner-space-01

From: Joe Touch <[email-masked]>
Date: Wed, 05 Nov 2014 16:55:52 -0800

Catching up...

On 10/27/2014 9:42 PM, Wesley Eddy wrote:
...
> For what it's worth, I also think the OOB approach is more desirable
> than the "dual-stack" type, however, am more uncertain about firewall
> behavior towards the segments with both SYN & ACK bits clear. 

Let's be clear - we're ALL in the dark about any of these, both as to
whether they will end up working or to what extent they would be
over-engineered to get around phantoms that don't exist.

> That's
> definitely a good experiment to get widespread data on.  I also don't
> really understand specifically how one decides how long to wait for
> the OOB segment before assuming it isn't coming,

You treat the OOB and SYN as a pair, and treat it with the same time-out
on both ends as far as giving up or retransmitting. How long you keep
queued copies to deal with reordering or other arrival-timing issues is
up to the receiver.

> but that could be
> tweakable or algorithmically determined, so I don't think it's a major
> issue.

Yes, esp. for the caching part.

Joe

---

From: Brian Trammell <[email-masked]>
Date: Thu, 6 Nov 2014 08:24:39 +0100

hi Joe, all,

I'm *almost* caught up on this thread, but I did want jump in to underline something that Joe said here:
 
> On 06 Nov 2014, at 01:55, Joe Touch <[email-masked]> wrote:
> 
> Catching up...
> 

+many.

One of the reasons it's been difficult to follow this whole discussion (beyond "I did not read email during the month of October") is that the model of the middle/meddle/muddleboxes we're trying to defend against is often implicit in the descriptions of the proposed mechanisms.

Further, it's not clear to me the extent to which these implicit models correspond to real-world devices, or to the extent that they do how often they are deployed in such a way that they would lead to reachability failure in the case of certain proposed extensions, or header/payload invariance failure that would lead to failure modes in those extensions. A middlebox that breaks a proposed extension mechanism in impossible to debug ways but that appears on 1e-7 of the paths on the Internet and is involved in 1e-14 flows falls into the category of "acceptable risk".

Further, even if we were arguing from explicit models backed up by openly available data, it would be unclear how the data from study A applies to situation B. Published network measurements relevant to these questions is subject to fairly large systematic biases, or put another way, we know much more about academic networks than we do about mobile access networks.

I see two ways forward here, which are probably complementary.

(1) Measure path impairment on the Internet

We pull together all the data from various studies of broken paths on the Internet, build from it a set of observations (i.e. at time t the path from x to y had these characteristics), look for correlations among those characteristics to allow the amplification of observations when possible, then correlate this "path impairment data" with estimates of traffic and topology information to allow us to quantify the risk of extension failure or extension-linked connectivity failure, and prioritize design effort based on this estimated risk. 

This is not a new idea. There is already a fair amount of work in this space, and probably a bit of infrastructure we could reuse, but to be useful for protocol design I think it would need to be placed into a single database, or at least various path impairment databases would need to share a common schema.

I recognize that this proposal has all the same problems with measurement data access that all other open measurement platforms do, with the added bonus that active measurements in this space will be interpreted as malicious by many of the same people who configure their firewalls to do end-to-end-hostile things. And given the people reading this message, even if we pulled all available data and designed lots of active measurements to plug the gaps, and all of you dropped everything you were doing to work on this, and told all your friends and colleagues to do the same, and they did so, we would still not have a very convincing answer to the bias question.

(2) Build path impairment measurement into the specifications and implementations of the extensions.

Or "fallback on steroids". This gives a tautological answer to the bias question, and if we're proposing vastly overengineering the extension mechanisms anyway we might as well *explicitly* have them measure in-line as part of the protocol interaction and/or simultaneously with protocol interactions in independent flows. 

Assuming one can devise a representation for path impairment that is useful and does not require the exposure of routable IP addresses or derived anonymized data from which routable IP addresses can be inferred, (2) can even be used as a source of data for (1) given the growing number of operating environments that already have a concept for "user consent to passive monitoring for debugging and development purposes" subject to the same magnitude of bias that (1) is already subject to.

Cheers,

Brian

> _______________________________________________
> tcpm mailing list
> [email-masked]
> https://www.ietf.org/mailman/listinfo/tcpm

---

From: Bob Briscoe <[email-masked]>
Date: Fri, 7 Nov 2014 19:26:03 +0000

Joe,

>queued copies to deal with reordering or other arrival-timing issues is

Not so IMO. The OOB has to be sent a 'guard time' after the SYN, to 
minimise the chance of it getting reordered and then getting tangled 
up in a firewall before the SYN arrives; the firewall will chuck any 
segment for which it has not seen a prior SYN.

So altho the timeouts might be the same it doesn't seem sensible to 
start the timers at the same time. Two possibilities:
a) the guard time is standardised.
b) the guard time is up to the client

If a), we (the IETF) don't know the operating conditions, so it's 
difficult to agree on a standard value. We can't calibrate it in 
units of RTT, because neither end knows the RTT at the stage when 
they are sending the SYN.

If b), the server end doesn't know what the guard time is, so its 
choice of when to start the timer for the OOB isn't straightforward.

I guess the data that is being collected by ALex Z & co /might/ help 
choose a guard time that avoids the worst re-ordering effects, 
whether it's chosen at standardisation time (by the IETF) or at 
run-time (by the client).

Bob

>Yes, esp. for the caching part.

________________________________________________________________
Bob Briscoe,                                                  BT

---

From: Joe Touch <[email-masked]>
Date: Fri, 07 Nov 2014 11:38:32 -0800

Hi, Bob,

...
> minimise the chance of it getting reordered and then getting tangled up
> in a firewall before the SYN arrives; the firewall will chuck any

Firewalls aren't the issue. NATs are, and unless reordering is prevalent
that won't be an issue.

Reordering would mostly happen on multiple paths, but you'd need
multiple paths between the client and the NAT - i.e., in the private
side. That's rare if it happens at all.

I thus don't agree that guard times are needed, which means we don't
need to worry about determining them.

> So although the timeouts might be the same it doesn't seem sensible to
> start the timers at the same time. 

I currently believe that there should be one timer. If you need to
resend, you resend BOTH the SYN and OOB. The server side can cache to
deal with reordering that happens there, but that doesn't require
guard-time at client transmission.

Joe

---

From: Andrew Mcgregor <[email-masked]>
Date: Fri, 7 Nov 2014 18:27:29 -1000

> side. That's rare if it happens at all.

It's rare right now, but there is lots of pressure to make reordering
protection in 802.11 aggregation optional, which will make it quite common
since I would expect it to be off by default on a lot of equipment with
firmware shipping late 2015 and thereafter.

> need to worry about determining them.

50ms would certainly do it, although that's more than many RTTs.  5ms would
be adequate in many cases.

-- 
Andrew McGregor | SRE | [email-masked] | +61 4 1071 2221

---

From: Joe Touch <[email-masked]>
Date: Fri, 07 Nov 2014 20:32:26 -0800

On 11/7/2014 8:27 PM, Andrew Mcgregor wrote:
> 
> 
> On 7 November 2014 09:38, Joe Touch <[email-masked]
> <mailto:[email-masked]>> wrote:
> 
>     Hi, Bob,
> 
> protection in 802.11 aggregation optional, which will make it quite
> common since I would expect it to be off by default on a lot of
> equipment with firmware shipping late 2015 and thereafter.

Reordering protection isn't the same as whether reordering will
typically happen.

However, we're talking about the packets with the same src/dst
addresses. Why would 802.11 aggregation affect their order? (I.e., I can
see it affecting the relative order of packets to/from different
destinations, but not the same ones)...

Joe

---

From: Andrew Mcgregor <[email-masked]>
Date: Fri, 7 Nov 2014 18:50:33 -1000

> destinations, but not the same ones)...

The 802.11 MAC can aggregate multiple IP packets into what's called an
AMPDU (and has to, in order to achieve decent performance).  Individual
packets within an AMPDU are checksummed separately, and can be
retransmitted individually or as part of the next AMPDU.  Right now,
reordering protection requires that IP packets (strictly, ethernet frames)
that have been correctly received but are behind (in link-layer sequence
number space) one that has not been received or explicitly NAKed have to be
buffered until receipt of the delaying frame or a NAK or a rather long
timeout.  Since the raw single-shot failure rate can be up to around 45%,
this happens a lot, and experiment indicates that if the traffic is
tolerant to a reasonable reordering window (as most current TCPs are, for
example) end-to-end performance is significantly better with reordering
protection disabled.

So yes, if reordering protection is off, packets within a single flow will
get reordered fairly frequently.

---

From: Joe Touch <[email-masked]>
Date: Fri, 07 Nov 2014 21:11:52 -0800

On 11/7/2014 8:50 PM, Andrew Mcgregor wrote:
...
> reordering protection requires that IP packets (strictly, ethernet
> frames) that have been correctly received but are behind (in link-layer
> sequence number space) one that has not been received or explicitly
> NAKed have to be buffered until receipt of the delaying frame or a NAK
> or a rather long timeout.  Since the raw single-shot failure rate can be
> up to around 45%, this happens a lot, and experiment indicates that if
> the traffic is tolerant to a reasonable reordering window (as most
> current TCPs are, for example) end-to-end performance is significantly
> better with reordering protection disabled.
>
> So yes, if reordering protection is off, packets within a single flow
> will get reordered fairly frequently.

The client can still get through a NAT a few different ways:

	- a gap (as suggested before)

	- bursting a small number of copies of SYN/OOB pairs
		the first SYN through a NAT would make way for
		a following OOB

First, though, tt'd be useful to understand what the error properties
are, esp. whether they're burst or uncorrellated.

And - with a single-shot failure rate that high, why FEC isn't being
considered.

Joe

---

From: Andrew Mcgregor <[email-masked]>
Date: Fri, 7 Nov 2014 19:21:58 -1000

On 7 November 2014 19:11, Joe Touch <[email-masked]> wrote:

> Indeed, those can work.

> are, esp. whether they're burst or uncorrellated.

Tend to be bursty, there are some studies on this so data is available.

> And - with a single-shot failure rate that high, why FEC isn't being

I don't know for sure, but if I had to speculate I'd say the answer would
be 'patents'.

---

From: Bob Briscoe <[email-masked]>
Date: Tue, 11 Nov 2014 04:31:54 +0000

Joe,

>Firewalls aren't the issue. NATs are, and unless reordering is prevalent

Just because NAT is an issue, doesn't mean a firewall won't be an 
issue as well.

>side. That's rare if it happens at all.

Once 2 back-to-back segments have got through a local NAT, they can 
get reordered over an access network (e.g. link aggregation group) 
before arriving at a firewall at the other end.

Also, TCP is used (and might be used in future) in rather more 
circumstances than just where a NAT is in the next room over a small LAN:
- There could be an access network (with link aggregation) then a 
carrier-grade NAT.
- The TCP client might be on a server in a data centre, opening a 
connection to a back-end database in another data centre over a WAN 
with ECMP in between.
- And so on...

Bob