Subject: Comments on draft-ietf-tcpimpl-pmtud-01.txt

From: Geert Jan de Groot <[email-masked]>
Date: Fri, 28 May 1999 00:58:11 +0200

I would like to make a couple of comments on the Path MTU Discovery 
Problems draft, and perhaps have a little discussion on it.

First, a quicky at the end of the document:

>3.3.   Determining MSS from PMTU
...
>     The MSS advertised at the start of a connection should be based on
      ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^
>     the MTU of the interfaces on the system.  Some systems use PMTUD
      ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^
>     determined values to determine the MSS to advertise.
>     This results in an advertised MSS that is smaller than the largest
>     MTU the system can receive.

I would like to change the marked sentence with:
The MSS advertised at the start of a connection should be based on
the largest MTU of all interfaces on the system.

Rationale: assume a TCP connect happens over an ethernet interface.
While the TCP session is alive, routing changes and the session is
now routed over an FDDI interface. If mss would have been 1460 (ethernet),
we would see micrograms on the FDDI interface unneccesary.
I just want to make the original text more explicit, that's all.

Then the real issue:
>3.1.     Black Hole Detection
...
>     Path MTU Discovery (PMTUD) works by sending out as large a packet
>     as possible, with the Don't Fragment (DF) bit set in the IP header.
>     If the packet is too large for a router to forward on to a particu-
>     lar link, the router must send an ICMP Destination Unreachable --
>     Fragmentation Needed message to the source address.
>
>     As was pointed out in [RFC1435], routers don't always do this cor-
>     rectly -- many routers fail to send the ICMP messages, for a vari-
>     ety of reasons ranging from kernel bugs to configuration problems.
>     Firewalls are often misconfigured to supress all ICMP messages.
>
>     PMTUD, as documented in [RFC1191], fails when confronted with this
>     problem.  The upper-layer protocol continues to try to send large
>     packets and, without the ICMP messages, never discovers that it
>     needs to reduce the size of those packets.  Its packets are disap-
>     pearing into a PMTUD black hole.

I would like to make things more explicit by adding another scenario,
one that I'm seeing every few weeks, and which IMHO warrants either
explicit mentioning in an RFC. This is a typical scenario:
- Many ISP's now run with MTU~4470 in their cloud, also on serial
  lines to their customers;
- An uncreasing number of webfarms apperently allow this large an MTU
  all the way to the web server;
- Assume a typical customer of such an ISP, using a Cisco box with
  Cisco-HDLC framing (which apperently doesn't catch MTU mismatches).
  Customer has a cache box with ethernet (to the Cisco), and FDDI
  (to the internal network), and hence MSS = 4352 - 40 = 4312 bytes.
- Note that the default MTU on Cisco serial lines is 1500 bytes.

If the customer surfs to such a high-MTU site, we see the SYN exchange,
the HTTP request gets sent, but data never arrives at the cache.
Investigation reveals that large packets are sent on the serial line
to the customer router, however because it's configured for smaller MTU,
this packet overruns the input buffer, so the packet is dropped
(and the giants error counter is bumped up).
It is important to note that, since the packet is not received correctly,
no ICMP message is sent, and therefore we have a blackhole scenario.

In defense I must mention that if the ISP sets up the customer router, 
the serial line is configured with the correct MTU, however chances
are that the customer sets up the router himself, the configuration
is lost at an upgrade or otherwise.

We see this kind of problem as a support case every few weeks; I have poked 
a bit around and there are numerous places where this potential problem
currently exists (but is masked in practive by a 1500-MTU hop
somewhere in the field).

>Implications
>     This failure is especially difficult to debug, as pings and some
>     interactive TCP connections to the destination host work.  Bulk
>     transfers fail with the first large packet and the connection even-
>     tually times out.

Note however, that it is possible to diagnose this by pinging with 
large packets: from the customer, try to ping the Cisco, then ping with
32K packets. If this all succeeds, repeat both tests with the ISP end
of the serial line. If the large packet ping test fails, you have the 
problem (though for completeness, the test should be run from both ends)

>     TCP should notice that the connection is timing out.  After several
>     timeouts, TCP should attempt to send smaller packets, perhaps turn-
>     ing off the DF flag for each packet.  If this succeeds, it should
>     continue to turn off PMTUD for the connection for some reasonable
>     period of time, after which it should probe again to try to deter-
>     mine if the path has changed.

I am not sure if I agree with this. First, there is a huge performance
impact; second, there is ambiguity between packets dropped because of
MTU problems and packets dropped because of congestion. The side effect
of the suggestion above is that congestion might lead to sending
micrograms unneccesary.
Second, this requires changes in the web server config, while it's
the _client_ path that's broken. I see no big incentive for webserver
maintainers to install this workaround if it becomes available.

So, my questions are:
- Is it appropiate to document the scenario above explicitely?
- Should it be documented in a tcpimpl document (it's not a tcpimpl issue!)
- I don't agree with the blackhole workaround, and believe it should
  not be recommended.

Comments?

Geert Jan

---

From: "Kevin Lahey" <[email-masked]>
Date: Thu, 27 May 1999 19:19:54 -0700

In message <[email-masked]>Geert Jan de Groot writes
>>     The MSS advertised at the start of a connection should be based on

Right, that's what was suggested in RFC 1191.  I changed NetBSD to 
work this way, and got my share of flack for it.  I'd love to see
every MSS advertised as 64KB;  if there is ever a TCPng, I'd hope
that MSS goes away.

I had the impression that this was a contentious issue in the past;
if no one objects, I'd be delighted to change it.

>[Black Hole Detection]

I'd be happy to add it, especially if lots of folks are seeing it.
Matt Mathis has mentioned similar scenarios.

>of the suggestion above is that congestion might lead to sending

It seemed like there was some consensus on this list that dealing
with the ambiguity was worthwhile if it meant that packets would 
continue to flow.

>maintainers to install this workaround if it becomes available.

Right, but the webserver maintainers are the ones who'd like the clients
to be able to receive the packets, and are the folks who can do something
about it.  The client side can only suffer and wonder at the delay...

>- I don't agree with the blackhole workaround, and believe it should

I'm not sure about the real answers, but now you've at least heard my 
best guesses.

Cheers,

Kevin
[email-masked]