Subject: [IPsec] Question about the draft-ietf-ipsecme-iptfs

From: Tero Kivinen <[email-masked]>
Date: Tue, 27 Apr 2021 17:13:50 +0300

While reviewing the draft for publication I found out that section 2.5
says that we first reorder packets, then make sure we have not missed
any packets, and only after that we process the in-order payload
streams to extract the inner-packets.

The problem is that packet is considered lost only when it falls out
of the re-ordering window.

This will cause huge delays in the traffic flows every single time we
loose even single outer packet.

I.e., if we have following flow (O<n> = outer packet n, I<n> = inner
packet n), assume packet bandwidth of 10 kB/s, 1400 byte packets,
re-order/replay window of 32 (typical for IPsec settings). This means
the packet send rate is 7.3 packets per second, and the size of
reordering buffer is 4.3 seconds:

Time	   Outer     Inner
0.000	   O<0>	     I<1>, pad
0.137	   O<1>	     pad
...
0.411	   O<3>	     pad
0.548	   O<4>	     I<2>, pad
0.685	   O<5>	     pad
...
2.055	   O<15>     pad
2.192	   O<16>     I<3>, pad
2.329	   O<17>     pad
...
3.014	   O<22>     pad,
3.151	   O<23>     I<4>, I<5frag1>
3.288	   O<24>     I<5frag2>, I<6frag1>
3.425	   O<25>     I<6frag2>, I<7>, pad
3.562	   O<26>     pad
...
4.247	   O<31>     pad
4.384	   O<32>     pad
4.521	   O<33>     pad

Now if the packet O<1> is lost, the algorithm requires us to wait
4.384 seconds (i.e, up to time 4.521), before we can forward I<2> -
I<7>, even when they were properly received several seconds before.

I think we should allow forwarding the inner packets which are
properly and completely received even when we have some earlier
packets missing.

This will of course also cause traffic to be very bursty in case of
packet loss in the inner packets, i.e., instead of getting inner
packets in same intervals they were sent, we suddenly receive all of
them after the first lost packet drops out of re-ordering queue, i.e.,
for the final recipient the timing will look like follows:

Time:
0.000	I<1>
4.384	I<2>
4.385	I<3>
4.386	I<4>
...

I.e., suddenly at 4.384 seconds final recipient will suddenly get
burst of back to back packets from the security gateway. This will
most likely cause all kind of issues on its own timing algorithms... 

If we do allow forwarding complete packets in out of order manner,
what should we do in case packet O<24> is lost? Should we allow
forwarding I<7> immediately when we received O<25>, or do we need to
wait 4 more seconds to see whether O<24> ever arrives before we deem
both I<5> and I<6> as lost (because fragments of them are lost), and
send I<4> and I<7> forward?

If we do allow processing of outer packets in that kind of out of
order manner, then that will of course mean that there might be
reordering happening because of this, and this most likely needs to be
mentioned in the draft too.

On the other hand I would assume that in normal cases lost packets are
much more common than reordered packets, but there are most likely
cases where there is lots of reordering, but not that much of lost
packets.
-- 
[email-masked]

---

From: Christian Hopps <[email-masked]>
Date: Tue, 27 Apr 2021 11:23:37 -0400

I'm not sure what you mean by Huge delays. Given an example of a 10 kilobit tunnel (really?) means *everything* is super slow, so then reporting raw wall clock times is a bit disingenuous I think. I didn't actually look over the math b/c this just doesn't seem realistic.

Have you considered what happens to your inner TCP stream bandwidth when you have regular packet loss, regardless of IPTFS? I think a slight bursty-ness given regular packet loss is the least of your problems at that point.

Thanks,
Chris.

---

From: Tero Kivinen <[email-masked]>
Date: Tue, 27 Apr 2021 20:52:14 +0300

Christian Hopps writes:
> I'm not sure what you mean by Huge delays. Given an example of a 10
> kilobit tunnel

10 kilobytes not kilobits, i.e. about 80 kilobits/s. This can handle
several low quality audio connections inside, or one high quality
uncompressed cd-quality mono channel.

> (really?) means *everything* is super slow, so then
> reporting raw wall clock times is a bit disingenuous I think. I
> didn't actually look over the math b/c this just doesn't seem
> realistic.

I do not really consider useful to waste bigger bandwidth for single
audio connection or similar, and I would assume audio connections is
one of those uses cases where proper TFC is useful.

Another use case could be terminal connection between servers (text
based ssh), I mean I can't really read xterm faster than one kilobytes
per second, so this provides 10 times the bandwidth needed for normal
terminal connection. I regularly have terminal connections between my
devices which lasts for weeks, and I would rather hide the fact when I
am actually using them, when I am not using them, but I do not want to
allocate tens of % of my home network bandwidth just for that case. 1%
would probably be acceptable (100 kilobits/s from the 10 megabits/s
inbound connection my VDSL offers).

Filetransfers, videostreams etc are not that interesting in the TFC
point of view, as they often try to use almost all bandwidth there is
(i.e., video stream will use as high quality than what your network
connection allows), and those usually have less of privacy concerns.

> Have you considered what happens to your inner TCP stream bandwidth
> when you have regular packet loss, regardless of IPTFS? I think a
> slight bursty-ness given regular packet loss is the least of your
> problems at that point.

Audio quite often uses UDP. Terminal connections do use TCP, and I do
regularly use SSH connections from all over the world to my home
server using RTT of 200-300 ms, and you do easily notice couple of
second delays when TCP looses packets. Now, when there would be
several seconds more just because TFC that would be even more
annoying.

What kind of bandwidth allocation you would do for audio connection
between two end-points? Lets say doorbell audio connection from your
door to your home over the mobile phone network? You might not want to
outsiders to notice when there is traffic between the gate and home,
or even when you start listening the sound from the gate...

Also note, that re-ordering and replay window size must be bigger than
largest re-ordering time, and sometimes when there are multiple
network links involved with bufferbloat etc, those times can be
measured in seconds, thus having replay window size of few seconds is
not unexpected configuration. This just mean that if you use 100
kbytes/s, i.e., 1 mbit/s bandwidth the replay window size will not be
32, it will be 256 packets, and if you use one megabyte / per second
bandwidth it will be 1000 packets etc. I think the 32 packets window
size is the minimal setting people use...

Of course, as I have been mostly working on the IoT environments and
standards lately my view of world is IoT centric, but those devices
are one of the devices which could benefit from the better security...
-- 
[email-masked]

---

From: Christian Hopps <[email-masked]>
Date: Tue, 27 Apr 2021 15:01:58 -0400

For very slow tunnels such as your indicating, you are not worried about out-of-order delivery; just set the reorder window to 0.

FWIW, the interest we are aware of is for 1GE to 100GE general purpose tunnels.

Thanks,
Chris.

---

From: Tero Kivinen <[email-masked]>
Date: Fri, 30 Apr 2021 18:58:21 +0300

Christian Hopps writes:
> 
> For very slow tunnels such as your indicating, you are not worried
> about out-of-order delivery; just set the reorder window to 0.

We do care about the replays even when we do not care about reorder,
so setting reorder window to 0 is not acceptable, as that would
effectively also set the replay window to 0, and this 

> FWIW, the interest we are aware of is for 1GE to 100GE general

I assume 1GE, means 1Gbit/s speed, i.e., with 1400 byte packets that
means about 100k packets per second, and with 1000 packet replay
window (which would be appropriate with that fast link), that means
each packet drop adds 1/100 s = 10ms delay until the reorder window
clears.

Anyways I think this should be mentioned in the draft, even if you do
not want to allow sending packets out as they come in, but delay and
buffer each packet until window clears.
-- 
[email-masked]

---

From: Christian Hopps <[email-masked]>
Date: Fri, 30 Apr 2021 12:18:26 -0400

The replay window does not need to be the same size as the reorder window.

Thanks,
Chris.

> On Apr 30, 2021, at 11:58 AM, Tero Kivinen <[email-masked]> wrote:
> 
> Christian Hopps writes:
>>