Subject: [RAM] Tunneling overheads and fragmentation

From: Robin Whittle <[email-masked]>
Date: Fri, 20 Jul 2007 00:34:27 +1000

Can anyone help me understand the likely problems with tunneling
packets and what MTU limits might be exceeded if LISP, eFIT-APT or
Ivip is used with IPv4?

I understand that fragmentation in IPv4 is very much to be
avoided.  I understand that in IPv6, each host is supposed to
explore how long it can make the packets without getting error
messages from some router en-route whose length limit is exceeded.

eFIT-APT tunnel endpoints are always within provider networks.
With LISP the ETRs and ITRs are also probably in provider
networks.  With Ivip, ITRs could be in the end-user network,
including an ITFH function in the sending host.

What sort of limits are there likely to be with the maximum length
of IP packets on any of these routes.

For instance, if all provider and transit routers happily handle
packets significantly longer than whatever hosts normally produce
(say 1500 bytes) then adding the encapsulation won't lead to any
fragmentation.  Is this a reasonable assumption?

What about routers in end-user networks, or CE routers of providers?

I could imagine some trouble if the main TCP/IP stack of a host
produced a packet which was at the maximum length which could be
handled by its DSL modem (and the NAT function of that
modem/router) but that if an additional item of software in the
host added an Ivip header to it, then there would be problems.

Likewise if the NAT function in the host had an ITFH function
which caused packets to get longer before they actually traversed
the "modem" link and went to whatever router was at the other end
of it, which might have an MTU limit which then gets exceeded.
Perhaps a NAT device with such a function would send out different
DHCP responses with smaller MTU and MSS so the hosts would
generally be making packets which were suitable for being
encapsulated.

In Mobile IP, how do mobile nodes handle the extra length of the
packets they tunnel to their home agent?  Does the additional
mobile IP software in the host operating system somehow change the
MTU and MSS of the layer above it which is preparing packets, not
knowing they are to be further encapsulated?

 - Robin

_______________________________________________
RAM mailing list
[email-masked]
https://www1.ietf.org/mailman/listinfo/ram

---

From: "Templin, Fred L" <[email-masked]>
Date: Thu, 19 Jul 2007 08:51:51 -0700

Robin,

This may not address all of your questions, but here is
something that may help with IPv6-in-IPv4 tunnel MTUs:

  http://www.ietf.org/internet-drafts/draft-templin-linkadapt-06.txt

See also RFCs 4459 and 4821.

Fred
[email-masked]

> -----Original Message-----
> From: Robin Whittle [mailto:[email-masked]] 
> Sent: Thursday, July 19, 2007 7:34 AM
> To: [email-masked]
> Subject: [RAM] Tunneling overheads and fragmentation
>

---

From: Stephane Bortzmeyer <[email-masked]>
Date: Thu, 19 Jul 2007 18:01:25 +0200

On Fri, Jul 20, 2007 at 12:34:27AM +1000,
 Robin Whittle <[email-masked]> wrote 

> packets and what MTU limits might be exceeded if LISP, eFIT-APT or

Executive summary: tunnels create a lot of problems (see the RFC
indicated by Fred Templin plus RFC 2923).

> fragmentation.  Is this a reasonable assumption?

I do not think so.

---

From: Iljitsch van Beijnum <[email-masked]>
Date: Thu, 19 Jul 2007 18:24:28 +0200

> I do not think so.

Agree, if you assume this, you'll get bitten.

However, I think it's a reasonable _requirement_. Hosts are connected  
by ethernet or PPP with an MTU of 1500 (or smaller) in nearly all  
cases, but it's just ethernet that lives in the 1970s with its 1500  
byte MTU but very little gigabit ethernet equipment sold this century  
can't do jumboframes to some degree.

It would also help to raise the internet MTU in general, of course:

http://www.ietf.org/internet-drafts/draft-van-beijnum-multi-mtu-00.txt

---

From: David Meyer <[email-masked]>
Date: Thu, 19 Jul 2007 09:30:33 -0700

> indicated by Fred Templin plus RFC 2923).

	The whole thing with PMTU discovery continues to come up
	whenever we debate tunneling. We spent a lot of time
	trying to document that particular issue in RFC 2784 (GRE). 

> > fragmentation.  Is this a reasonable assumption?

	Agreed. As mentioned above, there is also the interaction
	with PMTU discovery; basically, it can be nontrivial to
	find the original source of the packet so you can send a 
	a "fragmentation needed and DF set" back to the
	source. If indeed you can't find the source (multiple
	encaps or whatever), then the source's packets (those
	that have the DF bit set, i.e., doing PMTU discovery) get
	black-holed. Not sure if that was your question, but in
	any event...

	Dave

---

From: Wassim Haddad <[email-masked]>
Date: Thu, 19 Jul 2007 22:42:35 +0300 (EEST)

Hi Robin,

> knowing they are to be further encapsulated?

=> There is an ongoing work specific (as for now) to Mobile IP, which
removes the tunnel from the bidirectional mode and the overhead from
the route optimization. Please check: 
draft-haddad-mip6-tunneling-optimization-01

Regards,

Wassim H.

---

From: Robin Whittle <[email-masked]>
Date: Fri, 20 Jul 2007 13:18:49 +1000

Thanks for these great responses!

This message also concerns my argument for making the tunneled
packet's outer source address be the same as the inner source
address (outer SA = inner SA), as discussed in my second message
in the thread:  "ETRs checking src & dest addresses":

http://www1.ietf.org/mail-archive/web/ram/current/msg01705.html

The following is part of my To-Do list for Ivip:

  http://www.firstpr.com.au/ip/ivip/to-do/

Read these I-Ds and RFCs (I use these /html/ links because the
indicate later versions at the top of the page, and print with
proper page breaks):

  Link Adaptation for IPv6-in-(foo)*-in-IPv4 Tunnels

    http://tools.ietf.org/html/draft-templin-linkadapt-06

  MTU and Fragmentation Issues with In-the-Network Tunneling

    http://tools.ietf.org/html/rfc4459

  Packetization Layer Path MTU Discovery

    http://tools.ietf.org/html/rfc4821

  TCP Problems with Path MTU Discovery

    http://tools.ietf.org/html/rfc2923

  IPv6 Extensions for Multi-MTU Subnets

    http://tools.ietf.org/html/draft-van-beijnum-multi-mtu-00

  IP Tunneling Optimization in a Mobile Environment

http://tools.ietf.org/html/draft-haddad-mip6-tunneling-optimization-01

  Generic Routing Encapsulation (GRE)

    http://tools.ietf.org/html/rfc2784

> black-holed. Not sure if that was your question, but in

I didn't ask specifically, but it is a really important point:
Tunnels upset PMTU discovery.

Here are some diagrams of left-to-right packet flow, with the
outer header Source Address and Destination Address, and the inner
SA and DA for the part of the path where the original packet is in
a tunnel.

The first diagram is for the LISP/eFIT-APT approach of the outer
SA being that of the ITR.

           SH-----IR1----ITR~~~~~TR1~~~~~TR2~~~~~ETR----IR2-----RH

Outer SA       SH     SH     ITR     ITR     ITR     SH      SH
      DA       RH     RH     ETR     ETR     ETR     RH      RH

Inner SA                     SH      SH      SH
      DA                     RH      RH      RH

  ---- Raw
  ~~~~ Encapsulated

Now I consider where an "ICMP Destination Unreachable -
Fragmentation Needed" (DUFN) message might be generated.  I don't
know enough about this process to know if it is generated at the
output of an interface, and/or when arriving at an interface
(perhaps due to the innards of the router not being able to handle
this length).

If IR1 generates the DUFN message, all will be well - at least it
is not affected by the tunneling system.

Likewise if the ITR generates the DUFN when it receives the raw
packet.  But what if the ITR encapsulates the packet and then when
attempting to forward it, generates a DUFN?  Does the maximum
packet length vary from one outgoing interface to another?  How
will this part of the router know the problem is the
responsibility of a particular sending host and its own internal
encapsulation function?

If Transit Router TR1 generates the DUFN, the DUFN will be
addressed to the ITR.  But the ITR can't know what to do with it,
since as far as I know, the DUFN doesn't mention anything about
the contents of the packet, which is the only place where the
Sending Host's (SH's) address is mentioned.

Likewise if the DUFN arises from arriving at TR2 or the ETR.

If the DUFN is generated at the output of the ETR, it should be
OK, because it will be sent to the SH, not the ITR.  Likewise if
the DUFN is generated at the internal router IR2 or the
Destination Host (DH).

The best which could be achieved with the above arrangement is
that the ITR has to store state that packets sent to a particular
address (the ETR's address) should be fragmented before
encapsulation.  But how long does it retain this state for?  It
all depends on the routing path, since it could be a cranky old
TR2 which is complaining, and that may not be in the path for long.

Storing this state and burdening the ITR's FIB with looking for
it, for every encapsulation operation, and then splitting the
packet and sending two encapsulated packets . . . this is really
ugly.  Also, I am not sure how well LISP's or eFIT-ETR's
communications would work with packets being fragmented.  There
are some pretty fancy protocols for communication, particularly
between multiple ITRs and Default Mappers in eFIT-ETR. (See my
comparison message.)

Now consider the situation with Ivip's approach of setting the
outer SA the same as the inner SA.  This would be either with
IP-in-IP (which doesn't include the ITR's address) or with UDP
encapsulation with explicit mention of the ITRs address, if this
was decided to be worth the trouble to help with debugging - which
seems quite possible.

           SH-----IR1----ITR~~~~~TR1~~~~~TR2~~~~~ETR----IR2-----RH

Outer SA       SH     SH     SH      SH      SH      SH      SH
      DA       RH     RH     ETR     ETR     ETR     RH      RH

Inner SA                     SH      SH      SH
      DA                     RH      RH      RH

Now wherever the DUFN is generated, it will go back to the Sending
Host, which I think is what is required.

The ITR doesn't have to look out for DUFN messages, or store any
state, or do any optional fragmentation.

This way, the SA can fine-tune its packet size, per "RH"
destination address, to the highest value which avoids fragmentation.

Unless I have made some mistakes, then this looks like a second
strong reason for keeping "outer SA = inner SA", in addition to
the powerful reason (I think) which I raised in the thread "ETRs
checking src & dest addresses".

 - Robin

---

From: Gert Doering <[email-masked]>
Date: Fri, 20 Jul 2007 16:04:33 +0200

Hi,

> fragmentation.  Is this a reasonable assumption?

No.  As of today, too many links and routers still have issues handling
more than 1500 bytes of IP packets (like "most FastEthernet interfaces on 
Cisco routers", which is still considered "high bandwidth" over here).

> What about routers in end-user networks, or CE routers of providers?

Even worse, due to DSL encapsulation you already end up with an IP MTU
of 1492 bytes or less.

Gert Doering
        -- NetMaster
-- 
Total number of prefixes smaller than registry allocations:  113403

SpaceNet AG                        Vorstand: Sebastian v. Bomhard
Joseph-Dollinger-Bogen 14          Aufsichtsratsvors.: A. Grundner-Culemann
D-80807 Muenchen                   HRB: 136055 (AG Muenchen)
Tel: +49 (89) 32356-444            USt-IdNr.: DE813185279

---

From: Robin Whittle <[email-masked]>
Date: Sun, 22 Jul 2007 00:40:22 +1000

I just read:

  IP Encapsulation within IP
  http://tools.ietf.org/html/rfc2003

and the first few pages of:

I have added:

  IPv4 Reassembly Errors at High Data Rates
  http://tools.ietf.org/html/draft-heffner-frag-harmful-05

    IPv4 fragmentation is not sufficiently robust for use under
    some conditions in today's Internet.  At high data rates, the
    16-bit IP identification field is not large enough to prevent
    frequent incorrectly assembled IP fragments, and the TCP and
    UDP checksums are insufficient to prevent the resulting
    corrupted datagrams from being delivered to higher protocol
    layers. ...

to the To-Do list:

Gert's response and the responses of others makes me think that
these proposals - LISP-NERD/CONS, eFIT-APT and Ivip - cannot
succeed in their goal of requiring no changes to hosts, except
perhaps where host packet sizes are always controlled by DHCP
settings.

Even then, I fear that in order to preserve both reachability and
efficiency (and any reachability problems which arise from
fragmentation), that hosts in all networks, including non-upgraded
networks, will need to adopt a somewhat lower MTU setting - for
all the packets they send.

I wonder to what extent every possible application respects the
operating system's MTU.  Do operating systems simply refuse to
send any packet which is longer?  I wonder if there are any widely
used applications, such as games, P2P programs etc. which are
hard-coded to assume a certain MTU which is close to, or right at,
the limit of what can safely be sent across most of the Net.

Perhaps, in an optimistic scenario, recognising that some packets
are to be encapsulated by ITRs, an ISP network could set up its
DHCP system or whatever it is which gives DSL modems their
parameters, to reduce the MTU and MSS settings sufficiently that
the ITR-applied encapsulation still results in packets which will
not be fragmented in most transit and border routers.  I am
assuming a single level of encapsulation is all that is required.

Then, the hosts could generate marginally shorter packets - for
all packets sent, including those to non-mapped addresses and the
packets which need to be encapsulated will get to the other end
without any fragmentation.

This really needs to be done for all hosts in all networks - not
just hosts in networks which have been upgraded with ITRs and ETRs
etc.

Because there are no prospects of ramping BGP up to coping with
millions of advertised prefixes, every vaguely practical,
potentially incrementally deployable proposal for the routing and
addressing crisis involves tunneling.

I think the most likely way this fragmentation and MTU problem is
going to be solved is to back off the MTU and MSS settings on all
hosts, for all packets.  Even if the host had its own ITFH
(Ingress Tunnel Function in Host) function, I don't see how the
operating system could tell application programs that there is one
MTU and MSS setting for packets going to some addresses and
another setting for packets going to other addresses.

So unless someone figures out a totally different system, the only
way out of the current crisis will be to force all application
software on every host on the Net to generate somewhat shorter
packets.

If so, then I think this would be an argument in favor of the
shortest possible encapsulation system, which I think for IPv4 is
the IP-in-IP technique (RFC 3003), which adds 20 bytes.

UDP encapsulation, as used by LISP and I think eFIT-APT, involves
20 bytes for the IP header, 8 bytes for the UDP header and some
number of bytes, such as 4, for extra stuff which presumably
precedes the encapsulated packet itself.  For instance 32 bits of
length and other guff and a 32 bit address of the ITR, to identify
the ITR if my proposal for "outer SA = inner SA" is adopted.  So
that is 32 bytes of overhead.

This is where IPv6's long addresses and headers become really
ugly. There would be 40 bytes for IP-in-IP and 52 for basic UDP
encapsulation.

In my previous message:
    http://www1.ietf.org/mail-archive/web/ram/current/msg01729.html

I suggested a second reason (in addition to the original one of
making it easy to stop ETRs being used as a back-door around
filtering) for using "outer SA = inner SA": to allow ICMP messages
to go straight back to the sending host.  This would absolve the
ITR of the problematic, onerous or impossible task of receiving
ICMP messages coming back to it, and figuring out which sending
host to send back some ICMP message to.  (See RFC 4459.)

As far as I know, "outer SA = inner SA" is at odds with standard
practice, but I think it has some important benefits.

  - Robin

---

From: Marshall Eubanks <[email-masked]>
Date: Sat, 21 Jul 2007 10:43:56 -0400

>     some conditions in today's Internet.  At high data rates, the

I would change "some" to "most", at least as a practical matter.

Regards
Marshall

---

From: Robin Whittle <[email-masked]>
Date: Thu, 26 Jul 2007 11:01:14 +1000

Gert Doering wrote:

RW> For instance, if all provider and transit routers happily handle
RW> packets significantly longer than whatever hosts normally produce
RW> (say 1500 bytes) then adding the encapsulation won't lead to any
RW> fragmentation.  Is this a reasonable assumption?

In the thread "New LISP draft available ..." (msg01756), Dino wrote:

> All tunneling protocols have this problem. But I believe in
> practice it's not a real issue. Hosts send no larger, typically
> than 1500 byte packets and routers are now more and more on 9K
> MTU links.

These two statements don't seem to be reconcilable.

The further the ITRs and ETRs are from the sending and receiving
hosts, the less routers will be in the tunnel part of the path -
and the larger and newer those routers are likely to be, I guess.

For scaling purposes - to distribute the encapsulation and
decapsulation workload over more devices - we want to have more
ITRs and ETRs, which means they will be closer to the hosts (or
Customer Edge routers).  This is especially true for the ETR,
because as I argue in:

http://tools.ietf.org/html/draft-whittle-ivip-arch-00#section-14.1.2.5

ETRs must handle packets from ITRs in the same network

I think that packets to all LISP/APT/Ivip-mapped addresses sent by
hosts in the same network in which the hosts/CE-routers with the
destination LISP/APT/Ivip-mapped addresses reside . . . should be
forwarded via an ITR tunneling the packets to the right ETR,
rather than by the local routing system.  Only ITRs are likely to
be up-to-date on where the packets should be delivered to.

Otherwise (relying on the local routing system to forward raw
packets to the destination with the LISP/APT/Ivip-mapped address),
the local routing system has to have ITR functionality, which is
simple enough in Ivip - follow the database mapping - but very
complex for the other proposals.  I doubt the local routing system
could respond quickly enough to Ivip mapping changes, and I doubt
it could implement the functionality of LISP/eFIT-APT ITRs.

If this argument is accepted, then ETRs will be very close to the
destination, probably with a direct link - since they won't be
relying on the local routing system to get raw decapsulated
packets to the destination (unless they use a tunnel to the CE
router or the LISP/APT/Ivip-mapped host, which requires software
in the latter).

For scaling (load sharing) reasons, we probably want as many ITRs
as possible, which means they will be closer to the sending hosts
and that the larger encapsulated packets will flow through more
internal routers, which may be older (I guess) with lower
capacities and perhaps lower MTU sizes.

Gert also wrote:

RW> What about routers in end-user networks, or CE routers of
RW> providers?

> Even worse, due to DSL encapsulation you already end up with an
> IP MTU of 1492 bytes or less.

Putting an ITR function in a DSL modem (or HFC cable modem) NAT
device will require that the device use DHCP to give all hosts on
the private network a lower MTU so it can safely encapsulate the
packets which are addressed to mapped addresses, and not run into
MTU problems on its own DSL/cable link.

As far as I can see, this will lead to all application programs
being limited to somewhat lower packet lengths, unless somehow the
application (and operating system?) can be sure the packet is not
affected by LISP/APT/Ivip etc. mapping, and therefore is not going
to be encapsulated.

Is there a WG working on MTU issues, path MTU discovery etc.?

I have added this:

    tools.ietf.org/html/draft-van-beijnum-multi-mtu-00

to my To-Do reading list: http://www.firstpr.com.au/ip/ivip/to-do/

 - Robin