Subject: put on your PMTU discovery thinking caps

From: Vern Paxson <[email-masked]>
Date: Thu, 03 Dec 1998 23:43:57 PST

We've heard a number of people mention that it would be good to document
some of the pitfalls & problems of implementing PMTU discovery (something
with a bit more meat than RFC 1435).  For example, one is that if an ATM
switch doesn't have enough buffer to reassemble a bunch of cells into a
really large packet, then it will necessarily discard the packet, and if
the packet had DF set, it *won't* generate the necessary ICMP response
(because it couldn't process the full packet at all), leading to a PMTU
black hole.  (Matt Mathis pointed this one out a while back.)

Even though some of these problems aren't related to TCP implementation,
some are, and it's not clear there's a better home for discussing them than
tcpimpl.

So we've put on the Orlando agenda a discussion of PMTU discovery problems,
to try to get a feel for the scope.  Neither of your esteemed WG chairs,
though, is particularly familiar with these problems.  We'd therefore like
to encourage discussion on the list over the next few days, so we can get
things rolling in advance and figure out some of the common ground in order
to cover it quickly at the meeting.  We'd also like to encourage tcpimpl'ers
to come to the meeting prepared to discuss their favorite PMTU problems.

See you next week,

		Vern

---

From: "Kevin M. Lahey" <[email-masked]>
Date: Fri, 04 Dec 1998 13:10:21 -0800

In message <[email-masked]>Vern Paxson writes
>with a bit more meat than RFC 1435).

>to come to the meeting prepared to discuss their favorite PMTU problems.

I listed two small problems and one large (black hole discovery)
problem a few weeks ago.  Other than that, I guess I worry about the
oft-discussed issue of when to ACK -- if MSS really doesn't mean
anything anymore, do we ACK every two, or try to determine on-the-fly
the segment size in use by the sender.

An additional problem is MSS determination for multi-homed hosts.  
If we are responding to an SYN, we can use the MSS for the interface on 
which we received the SYN.  If we are originating a SYN, we can use the 
MSS of the interface out which we expect the packet to be sent.  What 
about routing changes?  If we belatedly find out that a connection
that can use our HIPPI interface (65280 byte MTU) instead of our
Ethernet interface, the MSS would prevent us from taking advantage
of the new PMTU.  We could use the MSS of the interface with the
largest MTU, but what happens we we plug in our HIPPI PCMCIA card :-)
half-way through a very long-term connection?

Kevin
[email-masked]

---

From: Geert Jan de Groot <[email-masked]>
Date: Tue, 15 Dec 1998 14:55:05 +0100

> with a bit more meat than RFC 1435).  

I have seen a particular failure mode a number of times; while it's
not an implementation fault, the failure mode is arcane enough that
it has lead to a number of escalations and it would be good to document it.

For example, see the following scenario:

          -------------          ----------
          |           |          |        | 
   (1)    |           |   (2)    |        |   (3)
 <========| ALG/proxy |----------| Router |-------->  ISP
          |           |          |        | 
          |           |          |        | 
          -------------          ----------

(1) is a FDDI ring to a customer's internal network (MTU=4470)
(2) is a DMZ existing of a regular ethernet 
(3) is a T1 serial link to an ISP.

There is a configuration error in this network: the MTU on the
serial line of the router is 1500 bytes (default); the ISP's end
of the serial line is 4470 bytes. The rest of the path to a remote
web server has MTU=4470 all the way, so that packets of this size
can be sent by the web server to the ISP end of serial line (3)
without being fragmented along the path.

Without PMTU discovery, this MTU mismatch would go undetected
as the proxy would send a small MSS (1460). Because of this, the web server
will send 1460-byte packets which will pass Router without a problem.

With PMTU however, the host MUST send a large MSS even over ethernet
(so that it can take advantage if a route flap causes the traffic
to go over FDDI, we could use FDDI's MTU). So, we'll send MSS=4430
and as a result, the web server will send 4470-bytes packets, 
which will be sent through the ISP's router onto the serial line.

However, because Router's serial interface is configured for MTU=1500,
the packet will get dropped because it's oversized. No ICMP error
message is sent because Router can't receive the whole packet
and hence can't generate an ICMP message.
Because no ICMP message is sent, packet retransmissions from the web server
will have the same size and will also be silently dropped.

This leads to the following scenario:
- One is able to ping remote site
- When sending a HTTP request (in this example), the TCP handshake
  happens, one is able to send the HTTP request (and have the request
  data ACKed by TCP), and then total silence till things time out,
  leading to responses like "the web site doesn't respond", or worse
  "this problem started with your new release with PMTU discovery,
  so your product has a problem with PMTU discovery"

Typically I detect this kind of problem by sending large-size ping
packets to the remote end of a suspect link. If the link is misconfigured,
then the ICMP echo response won't make it back to us (but it will work
for small-size ICMP echo packets). 
Traceroute won't be able to detect this because the response packet is 
always small.

I think that documenting these types of scenarios (which is similar to Matt's)
would be very useful. While it's indeed not in the direct scope of 
the tcp-IMPL wg, this venue does have the people with the right experience
so that a good document can be made.
I have done a little testing and found this type of problem more common
than I expected, so writing it up would be good.

Geert Jan

PS: I just saw a note in the is-is WG where they discussed making the
HELLO packet max size, just to detect his problem. I'm not sure if
putting this kind of test in a routing protocol is a good idea,
but having a widespread test would be good.