Subject: [tcpm] pmtu discovery (RFC 4821)

From: Thomas Narten <[email-masked]>
Date: Mon, 18 Apr 2011 10:47:58 -0400

Hi Matt.

What is the status of RFC 4821 uptake?

What is the current thinking w.r.t. 4821? Is it still considered the
Right Thing to implement? Or have implementation attempts concluded
its too complex to implement? Or?

The 6man WG is updating the Node Requirements document, and it was
suggested that it add a recommendation for 4821. The current RFC
(4294) was written before RFC 4821.

Thoughts?

Thomas

---

From: Matt Mathis <[email-masked]>
Date: Thu, 28 Apr 2011 08:37:05 -0400

Sorry I missed this earlier...

> its too complex to implement? Or?

I believe the Linux implementation for TCP is complete, but configured
for "black hole" detection only.   We wanted to configure it for
opportunistic jumbo detection, but there was an unanticipated driver
problem.  Basically the MTU/MRU has to be selected when the NIC is
initialized, before it can learn anything about the network.  If you
set the NIC MTU up, then you lower its performance on 1500 Byte
networks.

I was also told that it was on the tentative manifest for some now
past Microsoft release, but I have no idea if it actually made it.

> (4294) was written before RFC 4821.

I have always thought the IPv6 approach of mandating at least 1280
Bytes and everyone just presuming that the network was going to
magically deliver such a mandate was asking for troubles, especially
in the presence of tunnels, etc.  So yes I would require 4821 for
black hole detection.   The hard part (politically)  is you SHOULD
probe below 1280 bytes.

Thanks,
--MM--
The best way to predict the future is to create it.  - Alan Kay

---

From: rick jones <[email-masked]>
Date: Thu, 28 Apr 2011 08:39:28 -0700

> set the NIC MTU up, then you lower its performance on 1500 Byte

I like MTU's larger than 1500 bytes, and I may be mixing things  
incorrectly, but in the presence of TSO and GRO in the NICs isn't  
trying to find a larger MTU/MRU something of a diminished return?   
(IMO) The biggest bang from larger MTUs was the reduction of trips up  
and down the protocol stack, not the improvement in the ratio of data  
to data+headers.

rick jones

> _______________________________________________
> tcpm mailing list
> [email-masked]
> https://www.ietf.org/mailman/listinfo/tcpm

Wisdom teeth are impacted, people are affected by the effects of events

---

From: Joe Touch <[email-masked]>
Date: Thu, 28 Apr 2011 11:42:04 -0700

On 4/28/2011 8:39 AM, rick jones wrote:
...
> incorrectly, but in the presence of TSO and GRO in the NICs isn't trying
> to find a larger MTU/MRU something of a diminished return? (IMO) The
> biggest bang from larger MTUs was the reduction of trips up and down the
> protocol stack, not the improvement in the ratio of data to data+headers.

Larger MTUs reduces thrashing contexts in the up/down stack processing, 
and reduces the per-byte computation overheads. I don't think the 
returns are diminishing - going to 3000-byte packets halves the CPU 
processing for transport and network protocols, the interrupts or 
polling involved, etc.

I.e., it's not just about bandwidth overhead. Most protocol operations 
(except checksums, which are largely in hardware, and security 
protocols) are per packet, not per byte. This allows more data for a 
given header rate, which reduces system load for a given data capacity 
(or supports higher capacity, depending on how you view it).

Joe

---

From: Joe Touch <[email-masked]>
Date: Thu, 28 Apr 2011 13:25:27 -0700

Hi, Tim,

On 4/28/2011 1:13 PM, Tim Hartrick wrote:
>
>
> Joe,
>
> But....  Rick's point was that TSO and GRO have moved the non-checksum,
> per-packet protocol operations into hardware as well.  The consequence
> of this is that stacks now consume less CPU using 1500 byte MTUs with
> TSO and GRO than they would using jumbo MTUs without TSO and GRO.  It is
> certainly true using TSO and GRO with jumbo MTUs would reduce the load
> even further but at the cost of added complexity of detecting the
> presence of jumbo MTU capability.  Thus the reference to diminishing
> returns....

Per-byte operations in hardware are typically designed to support some 
existing data rate limit, e.g., PCI bus limits, link limits, etc.

Per-packet offload support is not always designed to those max rates. 
When they are, then there are diminishing returns for those external 
devices.

There are still operations that involve the CPU; when they are 
proportional to the number of packets, reducing the number of packets 
can help.

I think we're all on the same page; my points were for clarification, 
not disagreement.

Joe

> On Thu, 2011-04-28 at 11:42 -0700, Joe Touch wrote:
>>

---

From: Tim Hartrick <[email-masked]>
Date: Thu, 28 Apr 2011 14:13:31 -0600

Joe,

Tim Hartrick

---

From: John Heffner <[email-masked]>
Date: Fri, 29 Apr 2011 11:17:42 -0400

On Thu, Apr 28, 2011 at 11:39 AM, rick jones <[email-masked]> wrote:
> I like MTU's larger than 1500 bytes, and I may be mixing things incorrectly,
> but in the presence of TSO and GRO in the NICs isn't trying to find a larger
> MTU/MRU something of a diminished return?  (IMO) The biggest bang from
> larger MTUs was the reduction of trips up and down the protocol stack, not
> the improvement in the ratio of data to data+headers.

It feels like this discussion is going off on a tangent, but I'll note
there are still a few big wins a large paht MTU has over stateless
offload, especially on a WAN:

o Most congestion control algorithms operate on a per-segment basis,
and thus have an MSS term in the steady state throughput formula.  A
larger path MTU helps alleviate congestion control scaling issues.

o No current stacks I know of have a loss scoreboard implementation
that is O(1) on a per-segment basis.  (Most have constant lookup
optimisations for the typical case.  Some like FreeBSD bound the
maximum number of losses tracked, making it constant in theory as long
as you don't increase that bound.. but then you rely on timeouts at
larger window sizes.)  A larger path MTU makes the scoreboard more
efficient with large window sizes.

  -John

---

From: Caitlin Bestler <[email-masked]>
Date: Fri, 29 Apr 2011 09:39:40 -0700

> presence of jumbo MTU capability.  Thus the reference to diminishing

Whether or not a host uses hardware is irrelevant. It is still work that the host
has to do. A 63K tcp message is still more Ethernet frames when the PMTU
is 1500 than it is when the PMTU is 9K. Each of those frames needs to establish
at least the context of the TCP message, even if establishing the full context of
the TCP connection is deferred until the full message is received.

Now admittedly, if the message does not get interleaved with other traffic
then the cost of establishing the context of an already loaded TCP message
is exceedingly small. But it is something that must be done for LRO algorithms
whether that work is being done in hardware or a device driver.

So clearly a larger MTU will always benefit hosts, even if the benefit has been
properly limited to only restoring the context of a specific TCP message rather
than requiring the restoration of the context for the full TCP connection.

The real tradeoff is one of optimizing for the hosts versus the forwarding
elements. Larger frame sizes make efficient memory allocation in the
network more problematic. But nobody forces switches or routers to
support larger MTUs. I can't think of any reason not to take advantage
when working at L3 and above not to just take what L2 is willing to provide.

If any switch vendors think the standards are pushing larger MTUs to
the detriment of the network then they should make that argument.

But larger MTUs mean less work for IP hosts, wherever the host chooses
to do its work.

--
Caitlin Bestler
[email-masked]
http://www.asomi.com/CaitlinBestlerResume.html