Subject: [Plus] Blog post on quic and tou

From: Mirja Kühlewind <[email-masked]>
Date: Wed, 7 Dec 2016 16:12:16 +0100

hi all,

here is an interesting blog post on middleboxes:

https://www.snellman.net/blog/archive/2016-12-01-quic-tou/

Mirja

---

From: Jana Iyengar <[email-masked]>
Date: Wed, 7 Dec 2016 13:35:34 -0800

Thanks for forwarding the article. I'll offer some thoughts (and some
corrections.)

There's surely a solid argument to be made about network monitoring, as
this blog post makes. Operators' needs are real, and we need to ensure that
they are able to reasonably do the things that they need to do. At least
for QUIC, exposing a small additional bit of information addresses ~80% of
the use cases mentioned in the document: a "largest acked" ack number on
all packets (or at least packets that contain acks). I won't design this
mechanism on this list, but I'll note that it's a conversation that's
happening in several corners in the QUIC wg. It needs to be aired and
discussed, and I expect it to happen relatively soon.

The article though looks at real needs and current tools that operators
have, and over-generalizes to saying that the "entire header" should be
visible. My argument remains that only what is absolutely required should
be exposed, and that every bit exposed should be debated. This is not a
security argument, it's an ossification one.

The whole point of ossification is that there are third parties that are
unresponsive to changes in allegedly e2e protocols. Middleboxes are
reactive. If they see traffic shifting a particular way, they'll go build
something in response -- I've seen this happen several times. But, they are
not proactive. This creates a serious "deployment impossibility cycle"
where deploying a protocol change widely requires it to work through a huge
range of middleboxes, but even high-end middleboxes will not change
behavior in response until the protocol change is widely deployed. To the
extent that we want to be able to deploy end-to-end changes, unfortunately,
encryption is the only way to ensure that bits that ought not to be used by
a middlebox -- for fear of ossification -- should not be used by one. In Adam
Langley's words
<https://www.ietf.org/proceedings/90/slides/slides-90-httpbis-0.pdf>, "The
end-to-end principle is important, and cryptography is its strongest
guardian."

Having been deeply involved in both the TFO and QUIC efforts in Chrome, I
want to correct some things that the post mentions. First, even if they end
up taking the same amount of time to deploy, yield outcomes with
substantially different scales of benefit. After ~4 years of deployment
work, QUIC is now an open area for experimentation, but TFO is still
exactly one feature. Note that TFO is not yet deployed as widely as QUIC,
see Christoph Paasch's presentation at NANOG
<https://www.nanog.org/sites/default/files/Paasch_Network_Support.pdf> for
more.

On rollout:

"[1] There seems to be a bit of a difference in how this is rolled out. If
you want TCP Fast Open in Chrome, you'll need to enable it via a flag.
Meanwhile my understanding is that QUIC is effectively rolled out by
geographical regions; the flip that's getting switched is server side.
Presumably the latter rollout procedureincludes working with the main
service providers in that area to make sure any problems get fixed in
advance. A process like that would be a lot more tractable than trying to
fix the whole world at once."

This is incorrect. They were both rolled out using the exact same
framework. TFO and QUIC are both also available as flags for the user to
play with, but the experimentation framework is the same and rollout is the
same. Chrome pulled back TFO for a number of reasons. This is of course
still a WIP, and the hope is that we'll get TFO more deployed sooner than
later, but that hasn't happened yet.

On middleboxes:

"[0] Whoops, that's not quite accurate. There is one specific kind of
middlebox that a company like Google or Facebook needs: a load balancer.
And very conveniently, both protocols introduce a new field containing the
information a load balancer needs, and give it special treatment as the one
field that gets to live outside the encryption envelope."

The load-balancer needs to see the connection ID, but the load-balancer is
a trusted part of the serving infrastructure. Specifically, from a crypto
point-of-view, this load-balancer has access to various server keys. This
is quite different than an arbitrary middlebox. More importantly, from an
ossification perspective, a load-balancer ossifying would be equivalent to
the server ossifying.

- jana

On Wed, Dec 7, 2016 at 7:12 AM, Mirja Kühlewind <
[email-masked]> wrote:

> _______________________________________________
> Plus mailing list
> [email-masked]
> https://www.ietf.org/mailman/listinfo/plus
>

---

From: Brian Trammell <[email-masked]>
Date: Thu, 8 Dec 2016 10:35:59 +0100

hi Jana,

> On 07 Dec 2016, at 22:35, Jana Iyengar <[email-masked]> wrote:
> 
> Thanks for forwarding the article. I'll offer some thoughts (and some corrections.)
> 
> There's surely a solid argument to be made about network monitoring, as this blog post makes. Operators' needs are real, and we need to ensure that they are able to reasonably do the things that they need to do. At least for QUIC, exposing a small additional bit of information addresses ~80% of the use cases mentioned in the document: a "largest acked" ack number on all packets (or at least packets that contain acks). I won't design this mechanism on this list,

(...noting that this list exists *precisely* for designing this mechanism ;) but yes, details should happen on the quic@ list...)

> but I'll note that it's a conversation that's happening in several corners in the QUIC wg. It needs to be aired and discussed, and I expect it to happen relatively soon.

For those not familiar with the details of IETF-quic (which AFAIK from others' implementation reports diverges somewhat from the version of QUIC deployed by Google right now, and will diverge more during the WG's work): the packet number is already exposed. Together with highest-ack, this allows one-observation-point split-RTT measurement with an unknown responder delay term, equivalent to TCP; two-observation-point approaches for loss measurement; and one-observation-point approaches for loss estimation to work with more information about the dynamics of the particular version of QUIC running, also similar to TCP.

I personally think we can do a good deal better than this with epsilon more complexity and overhead, without either constraining QUIC's transport dynamics or requiring measurement devices to know about the details of those dynamics. Need to do a bit more work before I can say how small epsilon is, though.

Missing is more detailed information about TCP dynamics mentioned in the post. Many of these are TCP (and CC algorithm) specific, so it doesn't make much sense to expose the same information, though each of the requirements implicit in the list is worth evaluating separately for its ossification/security/utility tradeoff.

One requirement from that list that seems quite useful, though I don't know how to solve in the general case or in QUIC specifically: "determine [if] the software on the client or the server is the bottleneck". This is a very common triage task in network operations: does this problem indicate a misconfiguration of my network, or (more cynically) can I demonstrate that it's not my fault and therefore not my problem? Requiring access to one or both endpoint machines to answer that... seems like a question for a future Internet architecture research project.

> The article though looks at real needs and current tools that operators have, and over-generalizes to saying that the "entire header" should be visible. My argument remains that only what is absolutely required should be exposed, and that every bit exposed should be debated.

I would go further (again, this is the philosophical underpinning of PLUS, to the extent that it has one): the design of the header that is exposed unencrypted to the network (which constitutes its "wire image") should be treated as an entirely separate endeavour than the design of the transport protocol machinery.

> This is not a security argument, it's an ossification one. The whole point of ossification is that there are third parties that are unresponsive to changes in allegedly e2e protocols. Middleboxes are reactive. If they see traffic shifting a particular way, they'll go build something in response -- I've seen this happen several times. But, they are not proactive. This creates a serious "deployment impossibility cycle" where deploying a protocol change widely requires it to work through a huge range of middleboxes, but even high-end middleboxes will not change behavior in response until the protocol change is widely deployed.

My (possibly starry-eyed optimistic) hope is that a deliberately designed wire image will create a path of least resistance for middlebox designs to (reactively) follow. A well-designed wire image should be so obvious that the in-network reaction will be the one desired by the designers evenfor middleboxes built by people who didn't read the spec.

Cheers,

Brian

---

From: Jana Iyengar <[email-masked]>
Date: Thu, 8 Dec 2016 09:39:13 -0800

Hi Brian,

I'm largely in agreement; a couple of thoughts inline.

On Thu, Dec 8, 2016 at 1:35 AM, Brian Trammell <[email-masked]> wrote:

> mechanism on this list,
> (...noting that this list exists *precisely* for designing this mechanism
> ;) but yes, details should happen on the quic@ list...)
>
> > but I'll note that it's a conversation that's happening in several
> corners in the QUIC wg. It needs to be aired and discussed, and I expect it
> to happen relatively soon.
>
> For those not familiar with the details of IETF-quic (which AFAIK from
> others' implementation reports diverges somewhat from the version of QUIC
> deployed by Google right now, and will diverge more during the WG's work):
> the packet number is already exposed. Together with highest-ack, this
> allows one-observation-point split-RTT measurement with an unknown
> responder delay term, equivalent to TCP; two-observation-point approaches
> for loss measurement; and one-observation-point approaches for loss
> estimation to work with more information about the dynamics of the
> particular version of QUIC running, also similar to TCP.
>
> I personally think we can do a good deal better than this with epsilon
> more complexity and overhead, without either constraining QUIC's transport
> dynamics or requiring measurement devices to know about the details of
> those dynamics. Need to do a bit more work before I can say how small
> epsilon is, though.
>
> Missing is more detailed information about TCP dynamics mentioned in the
> post. Many of these are TCP (and CC algorithm) specific, so it doesn't make
> much sense to expose the same information, though each of the requirements
> implicit in the list is worth evaluating separately for its
> ossification/security/utility tradeoff.
>
> One requirement from that list that seems quite useful, though I don't
> know how to solve in the general case or in QUIC specifically: "determine
> [if] the software on the client or the server is the bottleneck". This is a
> very common triage task in network operations: does this problem indicate a
> misconfiguration of my network, or (more cynically) can I demonstrate that
> it's not my fault and therefore not my problem? Requiring access to one or
> both endpoint machines to answer that... seems like a question for a future
> Internet architecture research project.

This can be done by measuring at ingress and egress. A simplistic design,
at a high level, is:
(i) use packet number / ack number to measure queue buildup and loss
downstream of the ingress point, measured at the ingress.
(ii) use packet number / ack number to measure queue buildup and loss
downstream of the egress point, measured at the egress.
(iii) subtract (ii) from (i) to find queue buildup and loss in the network.

That at least gets you as far as identifying the problem as in the network
or outside it.

> be exposed, and that every bit exposed should be debated.
>
> I would go further (again, this is the philosophical underpinning of PLUS,
> to the extent that it has one): the design of the header that is exposed
> unencrypted to the network (which constitutes its "wire image") should be
> treated as an entirely separate endeavour than the design of the transport
> protocol machinery.
>
> > This is not a security argument, it's an ossification one. The whole
> point of ossification is that there are third parties that are unresponsive
> to changes in allegedly e2e protocols. Middleboxes are reactive. If they
> see traffic shifting a particular way, they'll go build something in
> response -- I've seen this happen several times. But, they are not
> proactive. This creates a serious "deployment impossibility cycle" where
> deploying a protocol change widely requires it to work through a huge range
> of middleboxes, but even high-end middleboxes will not change behavior in
> response until the protocol change is widely deployed.
>
> My (possibly starry-eyed optimistic) hope is that a deliberately designed
> wire image will create a path of least resistance for middlebox designs to
> (reactively) follow. A well-designed wire image should be so obvious that
> the in-network reaction will be the one desired by the designers evenfor
> middleboxes built by people who didn't read the spec.
>

The problem that remains is that once middeboxes react to certain bits,
deployment of changes to those bits requires herculean effort, as we see
with TFO. But this is exactly the tradeoff -- every bit that is exposed may
help the operator, but loses agility.

---

From: Brian Trammell <[email-masked]>
Date: Thu, 8 Dec 2016 21:40:05 +0100

hi Jana,

(inline, trimmed where appropriate)

> On 08 Dec 2016, at 18:39, Jana Iyengar <[email-masked]> wrote:
> 
> Hi Brian,
> 
> This can be done by measuring at ingress and egress. A simplistic design, at a high level, is:
> (i) use packet number / ack number to measure queue buildup and loss downstream of the ingress point, measured at the ingress.
> (ii) use packet number / ack number to measure queue buildup and loss downstream of the egress point, measured at the egress.
> That at least gets you as far as identifying the problem as in the network or outside it.

This requires a few properties, all of which would be satisfied by a QUIC that exports a packet number that always increments by one, even for zero-payload and packets with retransmitted frame (which I note it does unless I'm reading wrong), and exported an ack number that's really more of a "max packet number echo" (since the heuristics to make this work with TCP require you to ignore ACKs under loss conditions). (A delta term

Note that this is a two-point measurement methodology, which requires a far more elaborate measurement infrastructure than a one-point methodology, which basically just needs wireshark. The infrastructure requirements for diagnosis of a particular problem given a particular wire image are also a point to consider in the tradeoff space. (Or, more cynically again, as someone who's spent a lot of time building measurement infrastructures, thank you for pointing out a business opportunity. :) )

> The problem that remains is that once middeboxes react to certain bits, deployment of changes to those bits requires herculean effort, as we see with TFO. But this is exactly the tradeoff -- every bit that is exposed may help the operator, but loses agility.

Agree here completely. This isn't a "problem", rather it indicates a hard requirement to keep that wire image as minimal as possible.

Cheers,

Brian

---

From: Juho Snellman <[email-masked]>
Date: Fri, 9 Dec 2016 10:05:36 +0100

Hi,

On 8 December 2016 at 21:40, Brian Trammell <[email-masked]> wrote:
>
> Note that this is a two-point measurement methodology, which requires a
> far more elaborate measurement infrastructure than a one-point methodology,
> which basically just needs wireshark. The infrastructure requirements for
> diagnosis of a particular problem given a particular wire image are also a
> point to consider in the tradeoff space. (Or, more cynically again, as
> someone who's spent a lot of time building measurement infrastructures,
> thank you for pointing out a business opportunity. :) )
>

With measurements from all points in the network the encryption becomes a
non-issue anyway; we'd simply match up the packets between the traces. So
exposing information that's only useful in a multi-point context doesn't
seem all that compelling to me.

My experience is that there's a huge difference in the amount of effort
needed to get a single point trace vs. multiple traces. The first one has
basically no lead time, and can be just part of normal operating procedure
when investigating a problem report. The second tends to require careful
coordination across multiple groups, often a lead time of several days, and
gets routinely botched when traffic isn't being routed exactly as expected.
(I'm not aware of any easy to use tools for doing multi-trace analysis
either. That'd of course change if such tools become a necessity instead of
a luxury).

A packet number + ack number would indeed be the most useful thing to
expose, since it gives downstream RTTs and upstream packet loss. It doesn't
really do much for downstream packet loss though, which is a pretty
important component. For that, I think you'd want one of the following (in
order of preference):

- All the nack blocks
- The latest nack block
- The number of nack blocks
- A "this packet is a retransmission" bit in data packets. (Though this
would have other uses than just monitoring).
- A "there is at least one nack block" bit (this is weak enough that you'd
probably need a lot of heuristics)

After packet numbers, acks, and some kind of a reliable nack signal,
diminishing returns start to kick in, at least for the purpose of finding
network level problems.

I wouldn't be quite as cynical about client / server problems as you were,
though :) It's not simply a way to close a case as "not my problem"; it's
that it's easier and faster to prove a specific component is the problem
than to prove that the network as a whole isn't.

> Agree here completely. This isn't a "problem", rather it indicates a hard
> requirement to keep that wire image as minimal as possible.
>

Sure. When writing that post, I wasn't aware there was a discussion about
having more information outside the encryption envelope. So the contrast
was between "everything readable" vs. "nothing is readable". Of course I'd
always love to have more data available :) But if there's a sweet spot that
exposes just enough information, that'd be great.

-- 
Juho Snellman