Subject: Re: [IPsec] iptfs publication request

From: Christian Hopps <[email-masked]>
Date: Mon, 16 Aug 2021 03:59:24 -0400

Hi,

During the last IETF meeting it was agreed that we would move quickly to resolve any issues that Tero had left, and get this document submitted to the IESG. So asking again if the new version (published prior to the meeting) satisfied the issues so we can move forward?

Thanks,
Chris.

> On Jul 5, 2021, at 3:48 AM, Christian Hopps <[email-masked]> wrote:
> 
> 
> Christian Hopps <[email-masked]> writes:
> 
>> Hi Tero,
>> 
>> I understand you have some additional post-WGLC technical comments on the draft.  I'm sending this message to start a discussion thread so that the issue can be resolved and we can move forward on the publication request.
> 
> I have published a new version clarifying that one does *not* need to set the reorder and replay windows to the smaller of the two values, and in the case of slow tunnels it very well may make sense to have a larger replay window for logging purposes and a small or zero reorder window.
> 
> As this document achieved WGLC consensus over 3 months ago, I would request that the process please continue forward.
> 
> Thanks,
> Chris.
> _______________________________________________
> IPsec mailing list
> [email-masked]
> https://www.ietf.org/mailman/listinfo/ipsec

---

From: Tero Kivinen <[email-masked]>
Date: Mon, 16 Aug 2021 20:45:49 +0300

Christian Hopps writes:
> During the last IETF meeting it was agreed that we would move
> quickly to resolve any issues that Tero had left, and get this
> document submitted to the IESG. So asking again if the new version
> (published prior to the meeting) satisfied the issues so we can move
> forward?

No. You did not address any of my concerns there. I am not even sure
if you read my email where I presented my concerns, as I have not seen
any other comment on them than except complaining that this is not
meant for low bandwith situations and my examples using them were not
interesting.

Last week when I was reading your draft I got really disappointed when
I noticed that section 2.5 was completely unchanged, when you did say
you had addressed my comments. I got so annoyed, that I decided to
wait before sending my reply as I know how hard it is write
constructive text when you are annoyed.

You seemed way too much concentrating on my example using slow link
speed, and not to the fact that section 2.5 (implictly) requires you
to wait for packets to drop out of reorder window before they can be
processed at all:

Essentially the section 2.5 says that you first reorder, and wait for
all missing packets and finally processes the in-order payload streams
to extract the inner-packets.

>From my original email:

----------------------------------------------------------------------
If we do allow forwarding complete packets in out of order manner,
what should we do in case packet O<24> is lost? Should we allow
forwarding I<7> immediately when we received O<25>, or do we need to
wait 4 more seconds to see whether O<24> ever arrives before we deem
both I<5> and I<6> as lost (because fragments of them are lost), and
send I<4> and I<7> forward?

If we do allow processing of outer packets in that kind of out of
order manner, then that will of course mean that there might be
reordering happening because of this, and this most likely needs to be
mentioned in the draft too.

On the other hand I would assume that in normal cases lost packets are
much more common than reordered packets, but there are most likely
cases where there is lots of reordering, but not that much of lost
packets.
----------------------------------------------------------------------

I.e., I want section 2.5 to explictly mention how to do processing of
the AGGFRAG_PAYLOADS, i.e., is receiver allowed to forward parts of
the inner packets out even when previous packets are missing, and they
of course need to wait until all piecess of large fragmented packets
arrive before they can forward it, but can they send packets that were
in the first packet which did arrive out immediately when they arrive
(I would assume yes), and if some packets are missing in the middle, I
would assume that they can extract the inner-packets from later
packets too even when there is some packets missing.

The last paragraph of 2.5 which says:

   Finally, the receiver processes the now in-order AGGFRAG_PAYLOAD
   payload stream to extract the inner-packets (Section 2.2.3,
   Section 6.1).

would indicate that you do require full completele in-order payload
stream before you start extracting any inner-packets from the stream,
meaning that for every single lost or reordered frame you need to
stall until that frame drops out of reordering window to make sure
they have in-order payload stream. Only after the packet drops out
from the reorder window they can be sure it was lost thus only after
that they know they do have in-order stream that do have some gaps in
it.

So adding text saying "Receiver MAY process any completely received
inner-packets immediately they are fully received." would solve a lot
of problems, but of course that would contradict the last paragraph
which do say we process inner-packets only in order.

Also if we do allow processing inner-packets immediately when we have
received full packet inside the AGGFRAG_PAYLOAD payload stream this
may will cause the reordering which happened in the outer transport to
be duplicated for inner-packets too, so we should most likely add note
for that.

If we do not allow processing inner-packets immediately but must wait
until all the packets are received and the missing packet drops out of
reorder buffer that will cause us to create extra buffering and extra
burstines, so then we should add note about that.
-- 
[email-masked]

---

From: Christian Hopps <[email-masked]>
Date: Mon, 16 Aug 2021 14:51:47 -0400

> On Aug 16, 2021, at 1:45 PM, Tero Kivinen <[email-masked]> wrote:
> 
> Christian Hopps writes:

I believe I was focused on slow links b/c it seemed that you were focused on a concern of “large delays”, and the only large delays happen if there are slow links or inappropriately large reorder windows. In any case the revised text talks to these points.

> to extract the inner-packets.

We certainly did not want to specify whether an implementation SHOULD forward complete inner packets, out of order, or not, b/c that is not required for interoperability. Implementations could very well decide to offer one or both options. Our job is to specify the on-the-wire behavior so that implementations interoperate.

You are suggesting that we should add a “MAY” to the same affect though. That seems like a non-controversial change that could be made.

Adding a more text pointing out the obvious results of this choice  (i.e., that sending inner packets early can create downstream out of order delivery, or that waiting for outer packets can add delay and bursti-ness) would not be my preference. As a general rule I try (and I’m not alone in this) to avoid adding unneeded text, b/c the best you can hope for is to not say it wrong. That said if you believe the text must also include these 2 points as well i will add them in order to move things forward.

Would you be agreeable to just adding the clarifying “MAY” statement?

Thanks,
Chris.

---

From: Tero Kivinen <[email-masked]>
Date: Tue, 17 Aug 2021 00:09:39 +0300

Christian Hopps writes:
> We certainly did not want to specify whether an implementation
> SHOULD forward complete inner packets, out of order, or not, b/c
> that is not required for interoperability.

My reading is that current text says you MUST NOT forward complete
inner packets until all earlier packes have been received and you are
sure you have been able to sort the outer packets to in-order, and you
can only do that after the missing packets have dropped out of reorder
window.

> Implementations could very well decide to offer one or both options.
> Our job is to specify the on-the-wire behavior so that
> implementations interoperate.

I disagree on that. We also want implementations to do sane and smart
things, it is not just enough that they interoperate. Also specifying
recipient processing is good way to verify that your transmit
processing is also correct.

I am working also on the IEEE and they always say that they do not
specify receivers, they only specify transmitters, as implementing
receiver is left as exercise for the reader, and can simply be done by
"reversing" what transmitter does. This cause cases where they send
two variable length strings in one frame, without delimieters or
lenght fields and do not notice this as they only specify transmitter.
This is easy to implement on the transmiter end but impossible to
implement on the receiver...

Or in one case where their channel hopping sequence was selected by
calculating the hash of the packet. Very easy to do on the
transmitter as he has the packet. Bit harder to do on the receiver end
as he would need to be constantly trying all possible channel hopping
sequences and trying to see if any of them forms a packet...

I have always considered it very good that in the IETF we do try to
mandate or at least give instructions for the receiver end too. That
makes it much easier to do interoperable implemenations.

> You are suggesting that we should add a “MAY” to the same affect
> though. That seems like a non-controversial change that could be
> made.

Yes, I propose changing the MUST NOT to MAY...

> Adding a more text pointing out the obvious results of this choice
> (i.e., that sending inner packets early can create downstream out of
> order delivery, or that waiting for outer packets can add delay and
> bursti-ness) would not be my preference.

It might be obvious to you, but it might not be obvious to the person
doing the actual implementations. I always consider it a good idea to
point out pitfalls and cases where implementor should be vary to the
implementor and not to assume that implementor actually realizes this. 

> As a general rule I try (and I’m not alone in this) to avoid adding
> unneeded text, b/c the best you can hope for is to not say it wrong.

If you add unneeded text and find out later that it was then that was
good thing, as it clearly indicated that the text in question was not
clear enough. Adding more text that clarifies things is good thing. 

> That said if you believe the text must also include these 2 points
> as well i will add them in order to move things forward.

I think it would be worth explaining, as it might also cause the
implementor to think which option to choose. I.e., if he is more
concerned about the reordering, and do not care about the burstiness
or delays he will pick one option, but if he care more about the
latency he will pick another option. 

> Would you be agreeable to just adding the clarifying “MAY”

I think adding explaination about what the effects the two options for
MAY have is something that is good thing to add too.
-- 
[email-masked]

---

From: "Koning, Paul" <[email-masked]>
Date: Mon, 16 Aug 2021 21:22:47 +0000

> On Aug 16, 2021, at 5:09 PM, Tero Kivinen <[email-masked]> wrote:
> 
> ...

I agree with that sentiment.

The way I look at standards (protocol specifications in particular) is that conformance should imply interoperability.  If you give the spec to two people and ask them to implement what's in the spec, and they follow the stated rules, the result should be two implementations that interoperate.

	paul

---

From: Christian Hopps <[email-masked]>
Date: Mon, 16 Aug 2021 20:50:25 -0400

> On Aug 16, 2021, at 5:22 PM, Koning, Paul <[email-masked]> wrote:
> 
> 
> 

This is the specific case here:

“Given an ordered packet stream, A, B, C, if you send B before A you will be sending packets in a different order”

Again I’ll put this text to unblock this document, but really, sometimes things *are* obvious.

Thanks,
Chris.

---

From: Tero Kivinen <[email-masked]>
Date: Tue, 17 Aug 2021 13:48:52 +0300

> This is the specific case here:

No it is not.

> “Given an ordered packet stream, A, B, C, if you send B before A you
> will be sending packets in a different order”

The question there is that it is very hard to see from the text in
current draft section 2.5 that it can cause extra 32-1000 packets
(reorder window size) of buffering for every single lost packet.

And that current text does not allow the sending packets in different
order, as it does not allow processing packets in any other order than
in-order.

So there are multiple choises here, which affect how the
implementation behaves:

  1) Make sure that packets are processed in-order always, i.e., do
  not allow any outer packets to be processed until you are sure they
  are in-order thus causing extra buffering/latency if any packet is
  lost, as you need to wait for that packet to drop out of reorder
  window before you know it is lots, thus before you can continue
  processing packets. This will not cause any reordering of packets.

  2) Process incoming outer packets as they come in, and do not
  reorder them before processing. In that case you need to process
  outer packets partially, i.e., only send those inner packets out
  which have been fully received, but buffer those pieces of inner
  packets which are still missing pieces as outer packets were either
  lost or reordered. In this case if there is reordering in the outer
  packets this will cause this reordering on the inner packets too.

  3) Do hybrid version where when you notice missing packet on the
  outer packets you postpone processing of it for short duration and
  to see the reordering was only very small (for example wait for just
  next outer packet). If the outer packet stream can be reordered
  inside this small window you do that and process packets in order
  and send them out in order, but you limit the latency caused by this
  to for example only for one packet and if larger reordering is
  happening then you still buffer wait until the full reorder window
  until you deem that you are not able to process that inner packet as
  it was not completely received because of missing packet.

The current text only allows option 1, and I would like to allow
options 2 and 3 and perhaps also others, but I would also like to have
some text explaining the trade offs of different options. This does
not affect the interoperability as such, as two implementations using
different methods will interoperate, but this might cause very bad
performance issues.

Actually I think option 1 (the one only allowed now) can and will
cause large jitter for round trip times for every single lost frame. I
am not sure what large jitter of round trip times does for different
protocols running inside the tunnel. I would assume that any kind of
audio conferencing system would have really bad performance if run
over such system.

> sometimes things *are* obvious. 

I had to parse the section 2.5 several times before I realised that it
do really require me to process packets in-order i.e., it forbids
options 2 and 3.

It might be obvious for you, but it was not obvious for me, and I
think that restriction do make the performance really bad.
-- 
[email-masked]

---

From: Christian Hopps <[email-masked]>
Date: Tue, 17 Aug 2021 09:15:38 -0400

Hi Tero,

Let’s keep things simple here at this point in the process, and also match the results we have already verified with running code.

We can add more text that talks directly about how the reorder widnow size should be kept as small as possible (it should NEVER be 1000 packets, I’m not sure where you got 1000 from but that’s not a reasonable number so perhaps pointing this out IS important). It should be something between 0 and 5 perhaps 10 if you want to really handle wild cases of reordering (you probably dont).

Regular packet loss kills TCP etc.. we do not need to optimize the protocol for this condition; however, this brings me to the next point:

We are not transport experts here and we need to stay away from straying into that area. We had this draft reviewed and approved by the transport area (the experts) because of this. We should not start getting into transport area issues of describing affects on the network of jitter or re-ordered or lost packets etc. This is not our expertise and will only cause trouble when this comes before the transport area AD.

We have approved text from the transport experts now (in addition to clearing WG LC). I do not want to open this draft back up for major modifications that start talking about new ways to handle packets and their affects on the drownstream network etc. This is not our area of expertise, and we have already received approval from the experts for the text that we have. Let’s stick with the approved text and make clarifying modifications only.

Thanks,
Chris.

> On Aug 17, 2021, at 6:48 AM, Tero Kivinen <[email-masked]> wrote:
> 
> Christian Hopps writes:

---

From: Christian Hopps <[email-masked]>
Date: Tue, 17 Aug 2021 09:22:01 -0400

In particular why don’t we simply indicate that a lost packet can induce a delay of the fixed packet interval times the window size - 1, and so the widow size should be kept to a minimum, and leave it at that.

Thanks,
Chris.

> On Aug 17, 2021, at 9:15 AM, Christian Hopps <[email-masked]> wrote:
> 
> Hi Tero,
>

---

From: Christian Hopps <[email-masked]>
Date: Tue, 17 Aug 2021 09:37:56 -0400

I also need to point out that we are only talking about the case where the implementation doesn’t use a timer to timeout missing packets. We specifically added text highlighting that implementations are free to timeout missing packets much earlier if they so choose. Perhaps we should also highlight this again??

Thanks,
Chris.

> On Aug 17, 2021, at 9:22 AM, Christian Hopps <[email-masked]> wrote:
>

---

From: Tero Kivinen <[email-masked]>
Date: Tue, 17 Aug 2021 18:21:09 +0300

Christian Hopps writes:
> Let’s keep things simple here at this point in the process, and also
> match the results we have already verified with running code. 

In real production use over internet or in the lab between two test
machines (or inside one datacenter between machines there)?

> We can add more text that talks directly about how the reorder
> widnow size should be kept as small as possible (it should NEVER be
> 1000 packets, I’m not sure where you got 1000 from but that’s not a
> reasonable number so perhaps pointing this out IS important). It
> should be something between 0 and 5 perhaps 10 if you want to really
> handle wild cases of reordering (you probably dont).

Reorder and replay windows are linked as there is no point of having
replay window of 1000 packets if your reorder window is only few
packets, as any packets which gets delayed more than that few packets
which are accepted by replay window will be thrown away by reorder
window, thus there is no point of having bigger replay window than
reorder window.

Minimum replay window size required for each implementation is 32, and
I think that is very common default case, even though RFC4303
recommends window size of 64. From RFC4303 section 3.4.3:

   A minimum window size of 32 packets MUST be supported when 32-bit
   sequence numbers are employed; a window size of 64 is preferred and
   SHOULD be employed as the default.  Another window size (larger than
   the minimum) MAY be chosen by the receiver.  (The receiver does NOT
   notify the sender of the window size.)  The receive window size
   should be increased for higher-speed environments, irrespective of
   assurance issues.  Values for minimum and recommended receive window
   sizes for very high-speed (e.g., multi-gigabit/second) devices are
   not specified by this standard.

On fast links people want to increase that, as any packet which gets
delayed will get delayed a lot when counted in packets as even delay
of few tens of ms will mean there is lots of packets gone during that
time.

Actually I think it would be better to say that "replay window size
MUST be larger than reorder window size." and also add text that says
that if packet is out of reorder window (but was inside replay window
as otherwise it would have never reached re-ordering process) the
implementation can process the compelete inner packets from that
packets even when partial packets inside that outer packets cannot be
processed.

The replay window will take care that this is not an attack, so it is
safe to process such packets even when they are outside replay window.
Of course if it is outside replay window that means we can't
reassmeble partial packets that are part of that outer packet, but we
can safely process the inner packets completely inside that outer
packet. 

Adding text about that to the section 2.5 would also make the protocol
better. 

> Regular packet loss kills TCP etc.. we do not need to optimize the
> protocol for this condition; however, this brings me to the next
> point:

I do not belive so. I regularly use TCP over crappy mobile networks
and they do have lots of packet loss. TCP works fine in such
situations. It is not very fast, but it do work...

I have no idea how it would work when each packet loss would not be a
simple packet loss, but also a long delay before the further packets
in the stream are being received (i.e., waiting for the reorder buffer
to fill up before sending packets after the lot packet). I would
assume selective ACK and such mechanism in TCP will not really work,
but I am not expert, so I can't say that for sure.

This would require someone from the transport area to comment, how
well will TCP work when we randmly have extra 10x or 100x delay for
packet transition and then we get lots of packet back to back..

> We are not transport experts here and we need to stay away from
> straying into that area. We had this draft reviewed and approved by
> the transport area (the experts) because of this. We should not
> start getting into transport area issues of describing affects on
> the network of jitter or re-ordered or lost packets etc. This is not
> our expertise and will only cause trouble when this comes before the
> transport area AD.

True, but I am not sure if the transport area reviewer actually
understood the behavior we have specified now. I did not understood
that on my first reading.

> We have approved text from the transport experts now (in addition to
> clearing WG LC). I do not want to open this draft back up for major
> modifications that start talking about new ways to handle packets
> and their affects on the drownstream network etc. This is not our
> area of expertise, and we have already received approval from the
> experts for the text that we have. Let’s stick with the approved
> text and make clarifying modifications only.

When we find an issue in early phases of the document development
(WGLC is still very early in the document development), we should
address those. Also the section in 2.5 was written in such way that I
myself did not understand the implications of the words "Finally, the
receiver processes the now in-order AGGFRAG_PAYLOAD payload stream to
extract the inner".

I always assumed that we could process outer packets as they come in,
and did not have to do stop and wait until we have confirmed that we
have in-order stream (with perhaps some packets missing) before we
continue processing.

On the other hand it is true that we do get another transport are
review during the last call phase, and we can of course ask this
specific question from the reviewer at that time, but I think solving
this problem now is better than during the IETF wide last call.
-- 
[email-masked]

---

From: Tero Kivinen <[email-masked]>
Date: Tue, 17 Aug 2021 18:31:35 +0300

Christian Hopps writes:
> In particular why don’t we simply indicate that a lost packet can
> induce a delay of the fixed packet interval times the window size -
> 1, and so the widow size should be kept to a minimum, and leave it
> at that. 

As the window size is configured by the adminstrator and adminstrator
has no idea how it will affect things. We can't really expect
adminstrators to understand that when they have decide that they have
hundreds of GBs of memory in their VPN machines, that configuring the
reorder buffers and replay windows to 100000 packets long (absurdly
large value, but its just few gigabytes of memory for those buffers so
who cares about the memory consumption) that will suddenly cause
really long delays for all packets if any single packet is ever lost.

If the network is as reliable as you assume they always are and you
almost never loose packets it might take very long until this happens
and when it happens suddenly their whole network freezes until the
reorder buffer is filled up and then dumped to the network in one huge
burb.

This would be really bad behavior and most likely that would cause
them to submit bug reports to the implementations and implemetors
would blame the adminstrators for putting such stupid values in and
adminstrators would blame implementors for allowing such things and
not having warnings in the manual of not doing so, and implementors
said they did not realize that would happen as RFC did not say so.

If we do not explain why window size should be kept mimimum having
such text there does not mean anything. And I assume the usable
minimum would be the same than for minimum replay window size that
must be supported, i.e., 32 packet, which can still cause 32x of
normal delay for every lost packet.
-- 
[email-masked]

---

From: Tero Kivinen <[email-masked]>
Date: Tue, 17 Aug 2021 18:37:32 +0300

Christian Hopps writes:
> I also need to point out that we are only talking about the case
> where the implementation doesn’t use a timer to timeout missing
> packets. We specifically added text highlighting that
> implementations are free to timeout missing packets much earlier if
> they so choose. Perhaps we should also highlight this again?? 

I do not really see how this timer text helps, or at all related to
this discussion:

		Implementations that are
   concerned about memory use when packets are delayed (e.g., when an SA
   deletion is delayed), or non-IP-TFS uses of AGGFRAG mode, can of
   course use timers to drop packets as well.

It seems to cover cases where SA is deleted or non-IP-TFS uses of
AGGFRAG mode, which are not a concern here.

Or non-IP-TFS uses of AGGFRAG mode might be relevant here, but I think
the issues are also for IP-TFS uses of AGGFRAG. 
-- 
[email-masked]

---

From: Christian Hopps <[email-masked]>
Date: Tue, 17 Aug 2021 11:44:06 -0400

> I do not really see how this timer text helps, or at all related to

I'm saying we should add new text that mentions the use of this drop timer to drop missing packets after a short waiting time instead of just waiting for it to slide out of the reorder window. Then there is no issue to discuss anymore AFAICT.

Thanks,
Chris.

---

From: Michael Richardson <[email-masked]>
Date: Tue, 17 Aug 2021 12:18:34 -0400

Christian Hopps <[email-masked]> wrote:

    > induce a delay of the fixed packet interval times the window size - 1,
    > and so the widow size should be kept to a minimum, and leave it at
    > that.

Agreed.

    >> modifications that start talking about new ways to handle packets and
    >> their affects on the drownstream network etc. This is not our area of
    >> expertise, and we have already received approval from the experts for
    >> the text that we have. Let’s stick with the approved text and make
    >> clarifying modifications only.

I understand and agree.
Maybe clearly pointing at what text is involved would help.

--
Michael Richardson <[email-masked]>   . o O ( IPv6 IøT consulting )
           Sandelman Software Works Inc, Ottawa and Worldwide