Subject: Re: [lisp] Design discussion -06-(1) -> DF=1

From: Jesper Skriver <[email-masked]>
Date: Tue, 5 Jan 2010 14:55:53 +0200

On 4 Jan 2010, at 21:06, Templin, Fred L wrote:

> Jesper,
> 
>> -----Original Message-----
>> From: Templin, Fred L
>> Sent: Monday, January 04, 2010 7:06 AM
>> To: 'Jesper Skriver'
>> Cc: Dino Farinacci; [email-masked]
>> Subject: RE: [lisp] Design discussion -06-(1) -> DF=1
>> 
>> Hi Jesper,
>> 
>>> -----Original Message-----
>>> From: Jesper Skriver [mailto:[email-masked]]
>>> Sent: Monday, December 28, 2009 12:42 AM
>>> To: Templin, Fred L
>>> Cc: Dino Farinacci; [email-masked]
>>> Subject: Re: [lisp] Design discussion -06-(1) -> DF=1
>>> 
>>> Fred,
>>> 
>>> On 22 Dec 2009, at 18:24, Templin, Fred L wrote:
>>> 
>>>> Dino,
>>>> 
>>>>> -----Original Message-----
>>>>> From: [email-masked] [mailto:[email-masked]] On Behalf Of Dino Farinacci
>>>>> Sent: Monday, December 21, 2009 6:36 PM
>>>>> To: [email-masked]
>>>>> Subject: [lisp] Design discussion -06-(1) -> DF=1
>>>>> 
>>>>> A description of the design issue:
>>>>> 
>>>>> (1) Use stronger language to have the outer IPv4 header set DF=1 so we
>>>>>    can avoid fragment reassembly in an ETR or PETR.  This will also
>>>>>    make IPv4 and IPv6 encapsulation have consistent behavior.
>>>>> 
>>>>> Right now the spec does not recommend a preference on the DF setting
>>>>> of an outer IPv4 header. If a LISP encapsulator (a ITR or PITR/PTR)
>>>>> sets DF=0 an intermediate router could fragment the packet to the
>>>>> tunnel destination endpoint. That would be a ETR or PETR. We don't
>>>>> want to force ETRs and PETRs to reassemble packets if we can be
>>>>> practical. We want Path MTU discovery to be used on the path.
>>>>> 
>>>>> Do people have objections to recommending this?
>>>> 
>>>> The two dangers of doing this are that the ITR might not
>>>> receive legitimate ICMP "Packet Too Big" (PTB) messages,
>>>> and that it might receive spoofed PTBs. The ITR could
>>>> simply ignore all PTBs, but then there is danger of a
>>>> path MTU black-hole.
>>> 
>>> True, but the same is true for any other traffic sent with DF=1, which is a significant part of all
>>> traffic today. Also practical experience with deployment of IPsec based VPNs shows that path mtu
>>> discovery works well in practice.
>> 
>> RFC2923 among others shows that classical path MTU discovery
>> can have problems especially in the presence of encapsulation
>> when the effective MTU is reduced.

That RFC deals with interactions between TCP and path MTU discovery, which isn't relevant for this discussion, as LISP uses UDP for transport.

LISP using path MTU discovery between ITR and ETR, is independent from the hosts using path MTU discovery.

That said, path MTU discovery isn't fault free, but it is the best we got, and practical experience shows it to be working well - also it is mandated for IPv6, so why make LISP over IPv4 and IPv6 behave differently ?

>>> Indicating in the spec that path MTU discovery should be used, will ensure that it will be a rare
>>> event that an ETR or P-ETR will have to do reassembly prior to decap. Yet at the same time allow an
>>> ITR, which has problems receiving ICMP packet too big messages, to disable path MTU discovery for
>>> packets generated by it.
>>> 
>>> As it is the ITR which both controls if it sets DF=1 and it is the ITR which can have problems
>>> receiving ICMP packet too big messages, the control for solving any problem seen lies with the one
>>> that experience the problem, and thus we do not have a case where someone experiencing problems need
>>> to coordinate with a large number of other parties.
>>> 
>>> On the other hand, if we encourage packets to be sent with DF=0, and an ETR is connected via a lower
>>> MTU link that the typical ITR, then packets destined to that ETR with DF=0 will be typically be
>>> fragmented in the network before reaching the ETR. Forcing the ETR to do reassembly, without any way
>>> of preventing it, short of contacting the admin of any ITR which will be sending it traffic -
>>> something which clearly is not practical.
>> 
>> You are assuming that DF=0 means that the ETR will need to
>> reassemble if an MTU restriction is encountered - I am not.
>> I am proposing that the ITR sets DF=0 always, and that the
>> ETR listens for IP fragments. When the ETR receives an IP
>> first-fragment of a LISP packet, it sends a "packet too
>> big" message back to the ITR then systematically discards
>> all IP fragments of LISP packets that required fragmentation
>> *without performing reassembly*.

The ETR would only be able to identify the first fragment, as being a LISP packet, any subsequent fragments would not be identifiable, and as the ETR still needs to do reassembly for other IPv4 packets, these non-initial fragments would have to go into the fragmentation buffers, awaiting the initial fragment to arrive - which it never will.

I don't see this being practically implementable.

>> Upon receiving the "packet
>> too big", the ITR will reduce the  size of the packets it
>> sends, and the IP fragmentation will be tuned out.
> 
> Responding to my own proposal, rather than inserting any
> complicated machinery to discard fragmented LISP packets
> while reassembling others, the simplest solution would be
> to just set the ETR's reassembly buffer size to 576.
> 
> That way, any fragmented LISP packets would be discarded
> (since the unfragmented packet would almost certainly be
> larger than 576 bytes), yet RFC1812 which requires a 576
> byte minimum reassembly capability would be honored.

That would also break re-assembly of any other fragmented IPv4 packet received by the ETR, which I do not believe would be acceptable.

/Jesper

> 
> Fred
> [email-masked] 
> 
>> Fred
>> [email-masked]
>> 
>>> /Jesper
>>> 
>>> 
>>> 
>>>> 
>>>> Fred
>>>> [email-masked]
>>>> 
>>>>> Thanks,
>>>>> Dino/Darrel
>>>>> _______________________________________________
>>>>> lisp mailing list
>>>>> [email-masked]
>>>>> https://www.ietf.org/mailman/listinfo/lisp
>>>> _______________________________________________
>>>> lisp mailing list
>>>> [email-masked]
>>>> https://www.ietf.org/mailman/listinfo/lisp
>>> 
>>> /Jesper
>>> 
>>> 
>>> 

/Jesper

---

From: "Templin, Fred L" <[email-masked]>
Date: Tue, 5 Jan 2010 09:53:28 -0800

Jesper - see below:

> Sent: Tuesday, January 05, 2010 4:56 AM
> To: Templin, Fred L
> That RFC deals with interactions between TCP and path MTU discovery, which isn't relevant for this
> discussion, as LISP uses UDP for transport.

While the title of the document includes "TCP" and analyzes
the issues within the context of TCP, the issues are really an
IP-layer consideration and hence apply for any IP protocol
including UDP. Also, I did not intend the citation of RFC2923
as being the only such reference.  There have been numerous
other discussions of path MTU-related operational issues that
are readily available through a google search - especially
wrt interactions with encapsulation. 

> LISP using path MTU discovery between ITR and ETR, is independent from the hosts using path MTU

I think it may be more correct to say that the two are
interdependent. Yes, the LISP ITR discovers MTU restrictions
that occur within the tunnel. But, then it must relay
information regarding the restrictions back to the original
hosts. So, if the LISP ITR does not get path MTU feedback
then the original sources don't get it either.

> That said, path MTU discovery isn't fault free, but it is the best we got,

What I am suggesting is much better, however, and also simple
to implement.

> and practical experience
> shows it to be working well - also it is mandated for IPv6, so why make LISP over IPv4 and IPv6
> behave differently ?

IPv6 fixes some of the issues in that the ICMP PTB messages
are mandated to contain enough information for the ITR to
correlate the PTB message with the actual data packet that
triggered it. With IPv4, Fragmentation Needed messages need
only contain 8 bytes beyond the original packet header which
may not be enough for correlation or translation. But, with
both IPv6 and IPv4 path MTU discovery there is a reliance on
anonymous routers within the tunnel generating correct ICMP
messages and a reliance on the messages arriving at the ITR
without being dropped by a packet filter.

What I am suggesting avoids these dependencies because the
ETR is the only party that can generate the PTB messages;
none of the routers in the "black box" between the ITR and
ETR participate in the MTU discovery other than to perform
the mandatory IP fragmentation that is required of them.
Also, the PTB messages generated by the ETR simply look
like ordinary LISP packets so they should not be dropped
by any packet filters. 

> The ETR would only be able to identify the first fragment, as being a LISP packet, any subsequent
> fragments would not be identifiable, and as the ETR still needs to do reassembly for other IPv4
> packets, these non-initial fragments would have to go into the fragmentation buffers, awaiting the
> initial fragment to arrive - which it never will.
> 

> not believe would be acceptable.

OK, then let's not have the ETR mess with the reassembly
buffer size at all and just perform normal IP reassembly.
That way, if LISP packets from an ITR are being fragmented
the ETR may need to reassemble for a very short term while
its "Packet Too Big" messages are traveling back to the ITR.
But, once the ITR gets the PTB it will reduce the size of
the packets it is sending until the ETR no longer receives
any fragmented LISP packets.

However, the ETR should drop any LISP packets that arrived
as multiple IP fragments following reassembly. Otherwise,
packets that are too large to traverse the tunnel in one
piece may wind up making it through to the final destination
which may be confused into thinking that the path MTU is
larger than it actually is.

Fred
[email-masked]

---

From: Sam Hartman <[email-masked]>
Date: Mon, 11 Jan 2010 18:21:48 -0500

>>>>> "Dino" == Dino Farinacci <[email-masked]> writes:

    Dino> A description of the design issue: (1) Use stronger language
    Dino> to have the outer IPv4 header set DF=1 so we can avoid
    Dino> fragment reassembly in an ETR or PETR.  This will also make
    Dino> IPv4 and IPv6 encapsulation have consistent behavior.

    Dino> Right now the spec does not recommend a preference on the DF
    Dino> setting of an outer IPv4 header. If a LISP encapsulator (a ITR
    Dino> or PITR/PTR) sets DF=0 an intermediate router could fragment
    Dino> the packet to the tunnel destination endpoint. That would be a
    Dino> ETR or PETR. We don't want to force ETRs and PETRs to
    Dino> reassemble packets if we can be practical. We want Path MTU
    Dino> discovery to be used on the path.

I support recomminding/requiring the DF bit be set in the V4 header.

Based on our discussion of deployment model I think that we're really
going to have to take a look at PMTU issues.  I think this
recommendation will be consistent with our eventual results in this
space, but I don't think it will get us all the way there.

Fred and others have implied that PMTU discovery depends on getting ICMP
packet too big messages.  I'd like to remind people that the current
standards-track state of the art in this area (RFC 4821) actually deals
with lost ICMP messages.  The current LISP architecture is not rich
enough to support RFC 4821 between the ITR and ETR because it does not
have the right kind of probe packet support, although that would be a
relatively easy enhancement.

I'm surprised to find that RFC 5405 (particularly section 3.2) gives us
very little guidance in this area.

---

From: Dino Farinacci <[email-masked]>
Date: Tue, 12 Jan 2010 14:30:54 -0800

Here is proposed text to deal with this issue. Darrel will be posting  
to the list an entire diff file as well as the ID shortly.

Thanks,
Dino and Darrel