Subject: TCP Problems with Path MTU Discovery

From: Mark Allman <[email-masked]>
Date: Mon, 13 Sep 1999 22:02:28 -0400

Vern and I believe the pmtud I-D is nearing the point where we can
issue a last call and boot it up to the IESG.  However, there has
been very little discussion recently on the draft.  So, we'd like to
encourage folks to take a look at the draft and send any comments
you might have.  Unless there is a large amount of discussion we'll
probably issue the last call very soon.  The draft can be retrieved
via:

ftp://ftp.ietf.org/internet-drafts/draft-ietf-tcpimpl-pmtud-02.txt

Thanks,
allman

---

From: Geert Jan de Groot <[email-masked]>
Date: Thu, 16 Sep 1999 04:39:42 +0200

On Mon, 13 Sep 1999 22:02:28 -0400  Mark Allman wrote:
> issue a last call and boot it up to the IESG.  

I sent a list of comments on May 28 to tcp-impl but didn't see them
reflected in the -02 draft (nor responses that they shouldn't be
changed). I'm appending my original message below for your convienence.

I realise that the draft says 'configuration errors' but really would
like to make a case for listing them more explicitely, this is an
operational rather than an implementation matter and it just happens
too often...

Geert Jan

----

Date:    Fri, 28 May 1999 00:58:11 +0200
To:      [email-masked]
From:    Geert Jan de Groot <[email-masked]>
Subject: Comments on draft-ietf-tcpimpl-pmtud-01.txt

I would like to make a couple of comments on the Path MTU Discovery 
Problems draft, and perhaps have a little discussion on it.

First, a quicky at the end of the document:

>3.3.   Determining MSS from PMTU
...
>     The MSS advertised at the start of a connection should be based on
      ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^
>     the MTU of the interfaces on the system.  Some systems use PMTUD
      ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^
>     determined values to determine the MSS to advertise.
>     This results in an advertised MSS that is smaller than the largest
>     MTU the system can receive.

I would like to change the marked sentence with:
The MSS advertised at the start of a connection should be based on
the largest MTU of all interfaces on the system.

Rationale: assume a TCP connect happens over an ethernet interface.
While the TCP session is alive, routing changes and the session is
now routed over an FDDI interface. If mss would have been 1460 (ethernet),
we would see micrograms on the FDDI interface unneccesary.
I just want to make the original text more explicit, that's all.

Then the real issue:
>3.1.     Black Hole Detection
...
>     Path MTU Discovery (PMTUD) works by sending out as large a packet
>     as possible, with the Don't Fragment (DF) bit set in the IP header.
>     If the packet is too large for a router to forward on to a particu-
>     lar link, the router must send an ICMP Destination Unreachable --
>     Fragmentation Needed message to the source address.
>
>     As was pointed out in [RFC1435], routers don't always do this cor-
>     rectly -- many routers fail to send the ICMP messages, for a vari-
>     ety of reasons ranging from kernel bugs to configuration problems.
>     Firewalls are often misconfigured to supress all ICMP messages.
>
>     PMTUD, as documented in [RFC1191], fails when confronted with this
>     problem.  The upper-layer protocol continues to try to send large
>     packets and, without the ICMP messages, never discovers that it
>     needs to reduce the size of those packets.  Its packets are disap-
>     pearing into a PMTUD black hole.

I would like to make things more explicit by adding another scenario,
one that I'm seeing every few weeks, and which IMHO warrants either
explicit mentioning in an RFC. This is a typical scenario:
- Many ISP's now run with MTU~4470 in their cloud, also on serial
  lines to their customers;
- An uncreasing number of webfarms apperently allow this large an MTU
  all the way to the web server;
- Assume a typical customer of such an ISP, using a Cisco box with
  Cisco-HDLC framing (which apperently doesn't catch MTU mismatches).
  Customer has a cache box with ethernet (to the Cisco), and FDDI
  (to the internal network), and hence MSS = 4352 - 40 = 4312 bytes.
- Note that the default MTU on Cisco serial lines is 1500 bytes.

If the customer surfs to such a high-MTU site, we see the SYN exchange,
the HTTP request gets sent, but data never arrives at the cache.
Investigation reveals that large packets are sent on the serial line
to the customer router, however because it's configured for smaller MTU,
this packet overruns the input buffer, so the packet is dropped
(and the giants error counter is bumped up).
It is important to note that, since the packet is not received correctly,
no ICMP message is sent, and therefore we have a blackhole scenario.

In defense I must mention that if the ISP sets up the customer router, 
the serial line is configured with the correct MTU, however chances
are that the customer sets up the router himself, the configuration
is lost at an upgrade or otherwise.

We see this kind of problem as a support case every few weeks; I have poked 
a bit around and there are numerous places where this potential problem
currently exists (but is masked in practive by a 1500-MTU hop
somewhere in the field).

>Implications
>     This failure is especially difficult to debug, as pings and some
>     interactive TCP connections to the destination host work.  Bulk
>     transfers fail with the first large packet and the connection even-
>     tually times out.

Note however, that it is possible to diagnose this by pinging with 
large packets: from the customer, try to ping the Cisco, then ping with
32K packets. If this all succeeds, repeat both tests with the ISP end
of the serial line. If the large packet ping test fails, you have the 
problem (though for completeness, the test should be run from both ends)

>     TCP should notice that the connection is timing out.  After several
>     timeouts, TCP should attempt to send smaller packets, perhaps turn-
>     ing off the DF flag for each packet.  If this succeeds, it should
>     continue to turn off PMTUD for the connection for some reasonable
>     period of time, after which it should probe again to try to deter-
>     mine if the path has changed.

I am not sure if I agree with this. First, there is a huge performance
impact; second, there is ambiguity between packets dropped because of
MTU problems and packets dropped because of congestion. The side effect
of the suggestion above is that congestion might lead to sending
micrograms unneccesary.
Second, this requires changes in the web server config, while it's
the _client_ path that's broken. I see no big incentive for webserver
maintainers to install this workaround if it becomes available.

So, my questions are:
- Is it appropiate to document the scenario above explicitely?
- Should it be documented in a tcpimpl document (it's not a tcpimpl issue!)
- I don't agree with the blackhole workaround, and believe it should
  not be recommended.

Comments?

Geert Jan