Subject: [quicwg/base-drafts] Handling of corrupt Retry packets (#3014)

From: Kazuho Oku <[email-masked]>
Date: Tue, 10 Sep 2019 07:40:12 -0700

Retry packet does not have a checksum (e.g., CRC32, AEAD tag), and therefore is fragile to accidental bit flips that happen on the wire.

Are we fine with that? I ask this, because it seems to me that an one bit flip in some parts of a Retry packet can cause the connection establishment effort to fail in unexpected ways. To paraphrase, are we fine with certain deployments of QUIC being more fragile to accidental bit flips during connection establishment, than compared to TCP (that always have the checksum)? Note that checksum is an optional feature in UDP.

Below are what I believe would happen there is a bit flip in a Retry packet.

Consider the case where a bit in the SCID field of a Retry packet is flipped. The client would send back an Initial packet that carries this CID as the DCID. To the server, such an Initial packet would look like an attack, as it contains a Retry Token issued for a different server CID. Therefore, the server is likely to either drop the Initial packet, or close the connection immediately. This behavior would be a must for firewall generating Retry packets in order to redistribute the load between the servers running behind, as the load would be distributed by assigning different server CIDs to different servers.

Consider the case where a bit in the Token field of a Retry packet is flipped. The client would send back an Initial packet that carries this corrupt token. To the server, such an Initial packet *might* look like an Initial packet carrying an outdated token. Then, that server would send back a Retry packet carrying a new Retry token. But the client would drop the new Retry packet (because the client is required to accept only one Retry packet). The client continues retransmitting Initial packets using the corrupt token, until it time outs.

Bit flips in the DCID, ODCID, as such packets would be dropped by the client. Bit flips in the length fields are likely to cause no harm, and in the worst case scenario it would look either of the above.

PS. Version Negotiation packet also lacks a checksum. Though I think we are not going to use that packet type in V1 (see #3013).

-- 
You are receiving this because you are subscribed to this thread.
Reply to this email directly or view it on GitHub:
https://github.com/quicwg/base-drafts/issues/3014

---

From: Nick Banks <[email-masked]>
Date: Tue, 10 Sep 2019 07:47:21 -0700

One option would be to require UDP checksum for QUIC, right?

-- 
You are receiving this because you are subscribed to this thread.
Reply to this email directly or view it on GitHub:
https://github.com/quicwg/base-drafts/issues/3014#issuecomment-529970951

---

From: MikkelFJ <[email-masked]>
Date: Tue, 10 Sep 2019 07:57:13 -0700

@nibanks I don't think we should mix those layers. It would prevent QUIC from running elsewhere.

I do think that everything should be checksummed, like the initial packet. The problem is versioning, so each version need some official checksum schema.

-- 
You are receiving this because you are subscribed to this thread.
Reply to this email directly or view it on GitHub:
https://github.com/quicwg/base-drafts/issues/3014#issuecomment-529975592

---

From: Nick Banks <[email-masked]>
Date: Tue, 10 Sep 2019 07:59:03 -0700

I don't think it's a huge requirement that QUIC must be sent over a medium that is checksummed. IPv6 also has checksum already too, right?

-- 
You are receiving this because you are subscribed to this thread.
Reply to this email directly or view it on GitHub:
https://github.com/quicwg/base-drafts/issues/3014#issuecomment-529976463

---

From: MikkelFJ <[email-masked]>
Date: Tue, 10 Sep 2019 08:03:51 -0700

Well, sort of, but since QUIC in 99.999% if the cases does its own QUICsumming, it would be an odd requirement.

-- 
You are receiving this because you are subscribed to this thread.
Reply to this email directly or view it on GitHub:
https://github.com/quicwg/base-drafts/issues/3014#issuecomment-529978855

---

From: MikkelFJ <[email-masked]>
Date: Tue, 10 Sep 2019 08:04:31 -0700

Of course UDP sums still make sense for intermediate routing but that is fabric internal.

-- 
You are receiving this because you are subscribed to this thread.
Reply to this email directly or view it on GitHub:
https://github.com/quicwg/base-drafts/issues/3014#issuecomment-529979277

---

From: Nick Banks <[email-masked]>
Date: Tue, 10 Sep 2019 08:06:48 -0700

I'd personally prefer to depend on UDP in this case, especially since it already has performant hardware solutions. Why invent something new when there is a pre-existing solution? Also, we are only charted for UDP, as far as I know.

-- 
You are receiving this because you are subscribed to this thread.
Reply to this email directly or view it on GitHub:
https://github.com/quicwg/base-drafts/issues/3014#issuecomment-529980498

---

From: MikkelFJ <[email-masked]>
Date: Tue, 10 Sep 2019 08:08:37 -0700

Sure, only UDP, I'd just like to avoid hardcoding that dependency for a single case when you could just add a sha256 at the end or something.

-- 
You are receiving this because you are subscribed to this thread.
Reply to this email directly or view it on GitHub:
https://github.com/quicwg/base-drafts/issues/3014#issuecomment-529981354

---

From: ianswett <[email-masked]>
Date: Tue, 10 Sep 2019 08:10:09 -0700

I think it's reasonable to recommend using UDP checksums, particularly for unprotected packets such as Retry and VN.

I hadn't thought through the implications of a corrupted Retry token to realize it makes the handshake unrecoverable.  I'm not sure I want to fix that issue now, but I think it's worth documenting.

-- 
You are receiving this because you are subscribed to this thread.
Reply to this email directly or view it on GitHub:
https://github.com/quicwg/base-drafts/issues/3014#issuecomment-529982122

---

From: Kazuho Oku <[email-masked]>
Date: Tue, 10 Sep 2019 08:26:00 -0700

FWIW, I think that there are three options:
1. Document the behavior, recommend using UDP checksums.
2. Define a QUIC error code that recommends the client to retry establishing a connection. Then, a server recognizing the situations listed above can send that error code.
3. Add a checksum to the Retry packet. It could be a CRC, or could be a GMAC (i.e. AES-GCM without any encrypted data).

-- 
You are receiving this because you are subscribed to this thread.
Reply to this email directly or view it on GitHub:
https://github.com/quicwg/base-drafts/issues/3014#issuecomment-529989334

---

From: MikkelFJ <[email-masked]>
Date: Tue, 10 Sep 2019 08:58:55 -0700

Id suggest not using GMAC, at least for VNEG, because GMAC might be outdated before VNEG, and it is complex.

-- 
You are receiving this because you are subscribed to this thread.
Reply to this email directly or view it on GitHub:
https://github.com/quicwg/base-drafts/issues/3014#issuecomment-530004485

---

From: Kazuho Oku <[email-masked]>
Date: Tue, 10 Sep 2019 09:05:54 -0700

@mikkelfj Retry is a concept specific to QUIC v1, which already depends on AES-GCM for protecting the integrity of Initial packets. Therefore, IMO use of GMAC would be a reasonable choice _if we decide to give it a checksum_. Use of GMAC would be simpler than other algorithms, because then QUIC stacks can use the existing code that encrypts the Initial packet; Retry packet would be a packet that is encrypted exactly the same way as the Initial packet, with the only exceptions being that the nonce is fixed to zero and that the encrypted part is zero bytes.

-- 
You are receiving this because you are subscribed to this thread.
Reply to this email directly or view it on GitHub:
https://github.com/quicwg/base-drafts/issues/3014#issuecomment-530007372

---

From: MikkelFJ <[email-masked]>
Date: Tue, 10 Sep 2019 09:15:06 -0700

I don't think we disagree. I did mention VNEG explicitly. But, one could argue that that if VNEG uses a different hash, why not use it for Retry as well - not that I care that much.

-- 
You are receiving this because you are subscribed to this thread.
Reply to this email directly or view it on GitHub:
https://github.com/quicwg/base-drafts/issues/3014#issuecomment-530011189

---

From: Ryan Hamilton <[email-masked]>
Date: Tue, 10 Sep 2019 10:31:16 -0700

This definitely seems like a reasonable concern to me. I'd be inclined to extend to Retry packets the use of obfuscation/encryption similar to what we do for Initial packets. Heck, we could even encrypt the retry token itself if we really wished to (in keeping with our philosophy of obfuscating/hiding as much as humanly possible)

-- 
You are receiving this because you are subscribed to this thread.
Reply to this email directly or view it on GitHub:
https://github.com/quicwg/base-drafts/issues/3014#issuecomment-530041159

---

From: David Schinazi <[email-masked]>
Date: Tue, 10 Sep 2019 12:15:16 -0700

+1. Using AES-GCM-128 with initial salts for retry could solve this elegantly.

-- 
You are receiving this because you are subscribed to this thread.
Reply to this email directly or view it on GitHub:
https://github.com/quicwg/base-drafts/issues/3014#issuecomment-530080294

---

From: Kazuho Oku <[email-masked]>
Date: Tue, 10 Sep 2019 19:30:54 -0700

@nibanks 
> I don't think it's a huge requirement that QUIC must be sent over a medium that is checksummed. 

I am not sure. I doubt if it would be practical to suggest the could users to ask their could service provider "are you currently using UDP checksums, and committed to using that in the future?" Also, UDP checksums are rewritten by NAT devices, so there is no way to tell for an endpoint if a UDP packet is (was) protected by the UDP checksum.

-- 
You are receiving this because you are subscribed to this thread.
Reply to this email directly or view it on GitHub:
https://github.com/quicwg/base-drafts/issues/3014#issuecomment-530193705

---

From: Martin Thomson <[email-masked]>
Date: Tue, 10 Sep 2019 19:50:08 -0700

Presumably devices that rewrite addresses and checksums also check checksums on the way in...  Right?  :)

This seems like a fine design issue.  It's a fairly big change, for Retry, and late, but not impossible.  Version Negotiation is much, much more difficult to retrofit, so I think that we should concentrate on Retry here and open a VN issue if anyone feels strongly about that.

Using Initial packet protection isn't crazy.  We'd be able to drop ODCID if we did that.

Saying that we don't care is also rational.  The UDP checksum is imperfect (especially when it comes to catching bit flips), but we are talking about having maybe a few connections per day time out, globally.  I didn't do the numbers on that, but my recollection is that the odds are that small.

-- 
You are receiving this because you are subscribed to this thread.
Reply to this email directly or view it on GitHub:
https://github.com/quicwg/base-drafts/issues/3014#issuecomment-530197099

---

From: Ryan Hamilton <[email-masked]>
Date: Tue, 10 Sep 2019 19:57:36 -0700

Totally agree that VN is a different issue and this should focus on Retry.

I also love the idea of getting rid of ODCID! That has always seems like an ugly wart to me and replacing it with initial packet protection seems much more elegant. (As well as solving the packet corruption issue)

-- 
You are receiving this because you are subscribed to this thread.
Reply to this email directly or view it on GitHub:
https://github.com/quicwg/base-drafts/issues/3014#issuecomment-530198520

---

From: Jana Iyengar <[email-masked]>
Date: Mon, 16 Sep 2019 19:09:10 -0700

Jon Stone's work [[1](http://conferences.sigcomm.org/sigcomm/2000/conf/paper/sigcomm2000-9-1.pdf)] showed that the internet is actually quite terrible and that the TCP/UDP checksums are super important. From the conclusion section: "Our trace data shows that the TCP and UDP checksums are catching a signicant number of persistent errors. In practice, the checksum is being asked to detect an error every few thousand packets." Of course, this data is dated, but given the  I wouldn't be surprised if the numbers today were not too far off from the ones he reports.

@kazuho's right that we cannot rely on the UDP checksum being on in all circumstances, but I would guess that it'll be on in _most_ deployments. That said, the failure rate can be high enough to be bothersome in specific deployments. I would argue that it's a valuable goal to ensure that QUIC is protected against trivial corruption in all deployments.

I agree with @RyanAtGoogle that encrypting this packet is aligned with how we do things anyways, and there's additional benefit in being able to remove the ODCID. It is a bit late in the process, but this is a fairly isolated and local change.

I would argue that unless there's a strong reason to _not_ encrypt this packet, we should encrypt it. 

1: http://conferences.sigcomm.org/sigcomm/2000/conf/paper/sigcomm2000-9-1.pdf

-- 
You are receiving this because you are subscribed to this thread.
Reply to this email directly or view it on GitHub:
https://github.com/quicwg/base-drafts/issues/3014#issuecomment-532027451

---

From: Nick Banks <[email-masked]>
Date: Tue, 17 Sep 2019 06:31:35 -0700

I would prefer not to encrypt the Retry packet and keep things how they are. We've done a lot of work to create a design that requires the least amount of work to accomplish the stateless retry scenario. Adding an additional level of encryption, just to get a checksum (something that is practically, already accomplished in most deployments with UDP) is just going to add complexity (we've already started work with HW vendors on the current model, and it's complexity, compared to TCP, is already a major hurdle).

-- 
You are receiving this because you are subscribed to this thread.
Reply to this email directly or view it on GitHub:
https://github.com/quicwg/base-drafts/issues/3014#issuecomment-532221939

---

From: Dmitri Tikhonov <[email-masked]>
Date: Tue, 17 Sep 2019 07:43:36 -0700

To quote the paper @janaiyengar links to above:

> Traces of Internet packets from the past two years show that between 1 packet in 1,100 and 1 packet in 32,000 fails the TCP checksum

That translates to less than 0.1% error rate on the high side.  We already know that a significant percentage of QUIC connections will fail -- 3%, IIRC, according to Google -- for other reasons.  <0.1% is 3% of the 3%...  Is it worth it?

-- 
You are receiving this because you are subscribed to this thread.
Reply to this email directly or view it on GitHub:
https://github.com/quicwg/base-drafts/issues/3014#issuecomment-532252934

---

From: ianswett <[email-masked]>
Date: Tue, 17 Sep 2019 07:47:03 -0700

I think the concern here is not that the handshake fails, but that it takes a long time to fail in a circumstance where UDP connectivity does work.  Chrome currently uses quite a short handshake timeout of 4 seconds, but if clients use longer timeouts, then users could wait a long time in this unrecoverable state.

I think UDP checksums are probably sufficient here, but these long timeout based handshake failures are very user visible.

-- 
You are receiving this because you are subscribed to this thread.
Reply to this email directly or view it on GitHub:
https://github.com/quicwg/base-drafts/issues/3014#issuecomment-532254468

---

From: Dmitri Tikhonov <[email-masked]>
Date: Tue, 17 Sep 2019 08:00:49 -0700

@kazuho writes:
> Therefore, IMO use of GMAC would be a reasonable choice _if we decide to give it a checksum_.

This is an overkill.  Just use CRC32, which is simple, cheap (there is a dedicated CPU instruction for it!), and does what we need.

-- 
You are receiving this because you are subscribed to this thread.
Reply to this email directly or view it on GitHub:
https://github.com/quicwg/base-drafts/issues/3014#issuecomment-532260949

---

From: MikkelFJ <[email-masked]>
Date: Tue, 17 Sep 2019 09:02:04 -0700

Not arguing against CRC but intels CRC instruction is not necessarily standard CRC checksum for reasons that elude me.

-- 
You are receiving this because you are subscribed to this thread.
Reply to this email directly or view it on GitHub:
https://github.com/quicwg/base-drafts/issues/3014#issuecomment-532287119

---

From: Kazuho Oku <[email-masked]>
Date: Tue, 17 Sep 2019 09:02:36 -0700

@dtikhonov As people have been discussing, use of AES-GCM (or GMAC) is a simplification in terms of code reuse and also provides us the chance to get rid of the ODCID field. 

Clarification: GMAC is any-cipher-with-GCM with zero-byte encrypted text, so if it's going to be called AES-GCM or GMAC simply depends on if we decide to encrypt the token. If we are going to make the use of the QUIC-level checksum an optional feature, it would make more sense to not encrypt the token regardless of the QUIC-level checksum being used - which means GMAC.

-- 
You are receiving this because you are subscribed to this thread.
Reply to this email directly or view it on GitHub:
https://github.com/quicwg/base-drafts/issues/3014#issuecomment-532287312

---

From: Jana Iyengar <[email-masked]>
Date: Tue, 17 Sep 2019 13:50:20 -0700

@nibanks: Do you believe that obfuscating the packet is inconvenient at this late time, or that it is a lot of work for the HW vendors?

-- 
You are receiving this because you are subscribed to this thread.
Reply to this email directly or view it on GitHub:
https://github.com/quicwg/base-drafts/issues/3014#issuecomment-532396426

---

From: Nick Banks <[email-masked]>
Date: Tue, 17 Sep 2019 14:00:22 -0700

It's more about the cost per packet for the HW. With TCP, the devices handle millions, if not tens of millions of connections per second using the SYN cookie method. They won't be able to get anywhere close to that with QUIC Retry packets, and it will be even worse if every packet must be individually encrypted.

-- 
You are receiving this because you are subscribed to this thread.
Reply to this email directly or view it on GitHub:
https://github.com/quicwg/base-drafts/issues/3014#issuecomment-532400044

---

From: David Schinazi <[email-masked]>
Date: Tue, 24 Sep 2019 10:37:22 -0700

So far our design has been to take the processing costs in favor of making the protocol more reliable. Given that hardware will need to support sending encrypted Initial packets, I don't see the added cost on Retry as a blocking issue.

-- 
You are receiving this because you are subscribed to this thread.
Reply to this email directly or view it on GitHub:
https://github.com/quicwg/base-drafts/issues/3014#issuecomment-534666888

---

From: Nick Banks <[email-masked]>
Date: Tue, 24 Sep 2019 10:41:46 -0700

The HW I am referring to are standalone devices which only do the following:

1. Validate incoming Initial packets (primarily the Token field).
2. If valid, allow the packet through. If invalid, drop the packet and reply with a Retry packet.

No encryption or decryption of packets is necessary.

-- 
You are receiving this because you are subscribed to this thread.
Reply to this email directly or view it on GitHub:
https://github.com/quicwg/base-drafts/issues/3014#issuecomment-534668841

---

From: Martin Thomson <[email-masked]>
Date: Tue, 24 Sep 2019 16:34:39 -0700

Is there crypto involved in constructing a Retry token that can't be spoofed?

-- 
You are receiving this because you are subscribed to this thread.
Reply to this email directly or view it on GitHub:
https://github.com/quicwg/base-drafts/issues/3014#issuecomment-534788013

---

From: Nick Banks <[email-masked]>
Date: Tue, 24 Sep 2019 16:40:42 -0700

Two options exist:

1. Stateless - use crypto.
2. Stateful - generate random token and save in a cache.

I'm not sure which will be used in the final design. Even if we use crypto for the token, any addtional crypto for the packet is at least twice the CPU required to encrypt the token; halving the packets per second.

-- 
You are receiving this because you are subscribed to this thread.
Reply to this email directly or view it on GitHub:
https://github.com/quicwg/base-drafts/issues/3014#issuecomment-534789310

---

From: David Schinazi <[email-masked]>
Date: Tue, 24 Sep 2019 17:28:31 -0700

> Even if we use crypto for the token, any addtional crypto for the packet is at least twice the CPU required to encrypt the token; halving the packets per second.

This assumes that the great majority of packet processing time is spent on cryptographic operations, and that has never been my personal experience.

-- 
You are receiving this because you are subscribed to this thread.
Reply to this email directly or view it on GitHub:
https://github.com/quicwg/base-drafts/issues/3014#issuecomment-534799323

---

From: Nick Banks <[email-masked]>
Date: Tue, 24 Sep 2019 17:46:37 -0700

> This assumes that the great majority of packet processing time is spent on cryptographic operations, and that has never been my personal experience.

That may be so when doing things at the HTTP layer. I have had completely the opposite experience with my QUIC only performance tests. Encryption/decryption is closer to 60% to 75% of the CPU. UDP is the next largest, but because we've done a lot of optimizations in this area it's not that much.

But for HW, it's a whole different story. There is nothing but read QUIC header, read token, write token, write UDP packet. Read/write token will be, by far, the CPU bottleneck.

-- 
You are receiving this because you are subscribed to this thread.
Reply to this email directly or view it on GitHub:
https://github.com/quicwg/base-drafts/issues/3014#issuecomment-534802829

---

From: David Schinazi <[email-masked]>
Date: Tue, 24 Sep 2019 18:15:12 -0700

> There is nothing but read QUIC header, read token, write token, write UDP packet. Read/write token will be, by far, the CPU bottleneck.

I suspect that by "read/write token" you mean the compute/look-up token step, which I suspect won't be cheap, and that'll be there regardless of our decision on this issue.

I have a hard time believing that a hardware-accelerated AES-GCM step will take much longer than read/writing UDP. Making the protocol less reliable for the sake of a hypothetical performance result sounds like premature optimization to me.

-- 
You are receiving this because you are subscribed to this thread.
Reply to this email directly or view it on GitHub:
https://github.com/quicwg/base-drafts/issues/3014#issuecomment-534808512

---

From: Nick Banks <[email-masked]>
Date: Tue, 24 Sep 2019 18:26:08 -0700

UDP can gain huge perf boosts from batch processing. Crypto doesn't get the same boosts because each packet ends up needing a different key.

-- 
You are receiving this because you are subscribed to this thread.
Reply to this email directly or view it on GitHub:
https://github.com/quicwg/base-drafts/issues/3014#issuecomment-534811113

---

From: ianswett <[email-masked]>
Date: Tue, 24 Sep 2019 18:26:48 -0700

I agree with @DavidSchinazi that I have never seen profiles indicating that AES with standard(ie: AES-NI or AVX) acceleration is more costly than sending UDP packets.

I'm confused about whether you have an amazingly fast UDP path or a very slow AES-GCM path, relatively speaking.  I've seen CPU profiles where 50% of CPU is spent sending UDP packets on two different platforms now, but I've never seen a profile where AES-GCM consumed more than sending UDP packets.

Can you share any measurements that might explain the large disparity between CPU profiles I've seen and you've seen or shed any light on why our experiences are so different?

-- 
You are receiving this because you are subscribed to this thread.
Reply to this email directly or view it on GitHub:
https://github.com/quicwg/base-drafts/issues/3014#issuecomment-534811273

---

From: MikkelFJ <[email-masked]>
Date: Wed, 25 Sep 2019 00:15:56 -0700

A hardware forwarding UDP device has a completely different path from any general purpose OS, so it can't be compared at all.

AES as a number of roundtrips which requires a pipeline depth even if each step is fast and it might require custom specs if you have a fast simple core to start with.

Another option is to use good enough pseudo-crypto like some key mixed into a splitmix64 algorithm with frequently changing keys, at least for a part of the token field that affects the hardware filter.

-- 
You are receiving this because you are subscribed to this thread.
Reply to this email directly or view it on GitHub:
https://github.com/quicwg/base-drafts/issues/3014#issuecomment-534887912

---

From: Marten Seemann <[email-masked]>
Date: Wed, 25 Sep 2019 00:31:32 -0700

I agree with @kazuho, @DavidSchinazi and @ianswett. Making the protocol robust against bit flips should be our first priority, even if it's a (little?) bit more expensive for the sender of Retry packets.

-- 
You are receiving this because you are subscribed to this thread.
Reply to this email directly or view it on GitHub:
https://github.com/quicwg/base-drafts/issues/3014#issuecomment-534892744

---

From: Ryan Hamilton <[email-masked]>
Date: Wed, 25 Sep 2019 06:01:03 -0700

+1 Robustness is important.

On Wed, Sep 25, 2019 at 12:31 AM Marten Seemann <[email-masked]>
wrote:

> I agree with @kazuho <https://github.com/kazuho>, @DavidSchinazi
> <https://github.com/DavidSchinazi> and @ianswett
> <https://github.com/ianswett>. Making the protocol robust against bit
> flips should be our first priority, even if it's a (little?) bit more
> expensive for the sender of Retry packets.
>
> —
> You are receiving this because you were mentioned.
> Reply to this email directly, view it on GitHub
> <https://github.com/quicwg/base-drafts/issues/3014?email_source=notifications&email_token=AEVHVSQADYBUZAV4DKGKA6DQLMHVHA5CNFSM4IVITWZKYY3PNVWWK3TUL52HS4DFVREXG43VMVBW63LNMVXHJKTDN5WW2ZLOORPWSZGOD7Q5BSA#issuecomment-534892744>,
> or mute the thread
> <https://github.com/notifications/unsubscribe-auth/AEVHVSTU3D2AP3HSDX556PLQLMHVHANCNFSM4IVITWZA>
> .
>

-- 
You are receiving this because you are subscribed to this thread.
Reply to this email directly or view it on GitHub:
https://github.com/quicwg/base-drafts/issues/3014#issuecomment-535010726

---

From: Martin Thomson <[email-masked]>
Date: Wed, 25 Sep 2019 06:15:56 -0700

To be fair, this isn't just about running an AEAD over a relatively short packet.  It's running HKDF as well.  If you weren't inclined to process the packet, that's a fairly big jump in cost.

So it's not just twice the cost.  SHA-256 tends to be more expensive than AES, even with AES and its fairly clunky key setup.  You run multiple iterations of SHA-256 to generate the keys.

I personally think that this is worth doing, but I don't operate a service at scale.  I'd like something more concrete than "this is expensive", but I'm willing to concede that this is a considerable increase in cost for a DoS protection device.

-- 
You are receiving this because you are subscribed to this thread.
Reply to this email directly or view it on GitHub:
https://github.com/quicwg/base-drafts/issues/3014#issuecomment-535016758

---

From: Kazuho Oku <[email-masked]>
Date: Wed, 25 Sep 2019 06:28:05 -0700

> I'd like something more concrete than "this is expensive", but I'm willing to concede that this is a considerable increase in cost for a DoS protection device.

I share the same view. We have discussed this previously, and the conclusion was to not apply AEAD to the Retry packets. I think this issue warrants us the chance to revisit the previous conclusion, but I think it's worth wondering if we can find a solution without enforcing the use of AEAD.

That's why I am suggesting the use of optional GMAC. The straw-man is as follows.

When a client receives a Retry packet, it checks the last 16 bytes of the packet. If the last 16 bytes are equal to the original DCID, the packet is considered to be _not_ protected by the AEAD. The token is the rest of the packet payload. If the last 16 bytes is anything else, the packet is considered to be protected by the AEAD. The requirement to the server would be that it should either AEAD-protect the Retry packet, or use the UDP checksum.

There would no longer be a ODCID field.

I think this straw-man might be somewhat better than the "worst" of the two approach being combined, and that it might be something that we can go with.

-- 
You are receiving this because you are subscribed to this thread.
Reply to this email directly or view it on GitHub:
https://github.com/quicwg/base-drafts/issues/3014#issuecomment-535022144

---

From: Nick Banks <[email-masked]>
Date: Wed, 25 Sep 2019 07:31:19 -0700

I'm not exactly what more you need than "this is expensive". Obviously the HW can't be built and then we decide what to do. I just did a quick and dirty perf test (not sure if RSS was completely configured), and these were some numbers of our software QUIC stack for a CPU limited sender:
```
Encryption          45.32%
UDP Send            36.86%
Stream Data Copy     6.07%
ACK Processing       3.65%
```
The leftovers are for various smaller things. This is also on a machine that doesn't support all the HW offloads we support in Windows, so the UDP number can go down (drastically, giving numbers closer to the one I gave before). Also, while I am no crypto expert, I do not believe Windows bcrypt APIs are overly costly, compared to other libraries.

I know UDP has been expensive in the past, but that's no longer the case. I feel we should not use that past behavior as an excuse to default to computationally expensive solutions, where other solutions are available.

And again, like mikkelfj said you can't directly compare HW and SW solutions very well.

-- 
You are receiving this because you are subscribed to this thread.
Reply to this email directly or view it on GitHub:
https://github.com/quicwg/base-drafts/issues/3014#issuecomment-535051316

---

From: MikkelFJ <[email-masked]>
Date: Wed, 25 Sep 2019 07:51:30 -0700

So to be clear, I do vote a Initial like authentication scheme in Retry, as I hope I have stated earlier. But @nibanks concern is valid. I just don't think they have to be mutually exclusive:

You can have tail authentication tags and have other data that is fast for hardware to verify even if the filter is not perfect. I'd rather not rely on UDP headers for anything but IP routing.

-- 
You are receiving this because you are subscribed to this thread.
Reply to this email directly or view it on GitHub:
https://github.com/quicwg/base-drafts/issues/3014#issuecomment-535060731

---

From: ianswett <[email-masked]>
Date: Thu, 26 Sep 2019 12:59:55 -0700

If we want to give servers options, we could allow servers to send Initial packets with no payload and the client would treat those identically to a Retry.

-- 
You are receiving this because you are subscribed to this thread.
Reply to this email directly or view it on GitHub:
https://github.com/quicwg/base-drafts/issues/3014#issuecomment-535663360

---

From: Kazuho Oku <[email-masked]>
Date: Thu, 26 Sep 2019 13:30:49 -0700

@ianswett The benefit of tokens are that the state can be offloaded. Is that possible with Initial packets?

-- 
You are receiving this because you are subscribed to this thread.
Reply to this email directly or view it on GitHub:
https://github.com/quicwg/base-drafts/issues/3014#issuecomment-535674137

---

From: ianswett <[email-masked]>
Date: Thu, 26 Sep 2019 14:02:33 -0700

Functionally, a Retry or Initial with no payload should be identical, minus the packet format?
- Both have a Token
- Both have a way of validating the original CID
- Both have no other payload

I believe we considered using the Initial packet as a Retry packet in the Stream0 design team, but the reason we decided not to was some members thought the encryption cost was too high.  It looks like we're still in the same place in that regard?

-- 
You are receiving this because you are subscribed to this thread.
Reply to this email directly or view it on GitHub:
https://github.com/quicwg/base-drafts/issues/3014#issuecomment-535684862

---

From: ianswett <[email-masked]>
Date: Thu, 26 Sep 2019 14:13:41 -0700

@nibanks Those numbers are very different from the ones I've seen.  In particular, I've never seen UDP send lower than encrypt that I can recall.

I will note that I'm typically looking at profiles of a full process(ie: a load balancer/cache/etc), not just the QUIC stack, but even so your crypto is much more expensive than I'd expect.

Are you using AES-NI and/or AVX instructions?

-- 
You are receiving this because you are subscribed to this thread.
Reply to this email directly or view it on GitHub:
https://github.com/quicwg/base-drafts/issues/3014#issuecomment-535688505

---

From: Nick Banks <[email-masked]>
Date: Thu, 26 Sep 2019 14:24:24 -0700

Yes, Bcrypt APIs make full use of CPU instruction offload where available. As I indicated, these numbers are from a QUIC layer performance test (no HTTP involved) between two machines on a 40Gbps switch (i.e. unlimited bandwidth). The machines are CPU limited. UDP CPU costs are greatly reduced by batching many packets worth of data in a single send call. HW offloads improve this even more.

-- 
You are receiving this because you are subscribed to this thread.
Reply to this email directly or view it on GitHub:
https://github.com/quicwg/base-drafts/issues/3014#issuecomment-535691886

---

From: Kazuho Oku <[email-masked]>
Date: Thu, 26 Sep 2019 18:36:37 -0700

@ianswett 
> * Both have a Token
> * Both have a way of validating the original CID
> * Both have no other payload

Ah thank you for the clarification. So the idea here is to allow the servers to send tokens in Initial packets (which is currently prohibited), when the payload length is zero.

I agree that that's a possibility, and that we could have taken that path.

OTOH, I am not sure if we should consider that at this late stage, even if we are to authenticate and encrypt the Retry packet. The reasons are:
* Use of the Initial packet does reduce the complexity. It just changes how the clients make the distinction between Initial vs. Retry.
* The existence of PN field is concerning, as the client cannot ACK a Retry.

>From the server's point of view, Retry is an action that happens before a connection is established, and I like the fact that it is given a distinct packet type rather than being defined as a subtype of a packet that could be part of a connection (i.e. carrying PN). And we have enough bits to express the four long header packet types.

-- 
You are receiving this because you are subscribed to this thread.
Reply to this email directly or view it on GitHub:
https://github.com/quicwg/base-drafts/issues/3014#issuecomment-535746870

---

From: ianswett <[email-masked]>
Date: Fri, 27 Sep 2019 07:38:24 -0700

Thanks @kazuho I forgot Retry didn't have a PN.  That is a bit concerning.

-- 
You are receiving this because you are subscribed to this thread.
Reply to this email directly or view it on GitHub:
https://github.com/quicwg/base-drafts/issues/3014#issuecomment-535967378