Subject: [v6ops] proposed TCP MSS text for rfc6204bis

From: "Hemant Singh (shemant)" <[email-masked]>
Date: Mon, 14 May 2012 10:13:23 -0500

[The CPE router MUST support a TCP MSS Adjust feature on packets
traversing the CPE router.  By default, the TCP MSS Adjust feature is
turned off. ] 

Some thoughts below on why the above text.

The TCP MSS Adjust is not needed for TCP packets sourced or destined to
the CPE because the number of such packets at the CPE will be very
small.  Additionally, we (Wes and I) deliberately left out qualifying
the feature for IPv4 or IPv6 because the feature should be supported for
both since the CPE also supports 6rd.   We have left out specifying any
default value for the MSS because there are several values since the MSS
is a function of the protocols being used.   Note IPv4 CPE routers
already support a TCP MSS Adjust knob.    Further, the DS-Lite AFTR and
the 6rd BR in the SP domain sit between the home TCP client and a TCP
server on the Internet.  Thus the AFTR and the BR can perform TCP MSS
Adjust.  It's only for 6rd  where the CPE's talk without going through
the BR, the CPE has to invoke the TCP MSS Adjust.  

The problems that the above text alleviates are:

(a)    Any of native IPv6  and tunneled technologies such as DS-Lite and
6rd can cause ICMPv6 errors for packet too big to the source.   Even
when the CPE issues the ICMPv6 error to the host connected to the CPE,
the Internet access of the host is delayed which is not good.
Additionally, what if the CPE passes the host packet to the Internet and
one router on the Internet issues the ICMPv6 error with packet too big
but a node in the path back blocks the ICMPv6 error.   Now the Internet
connectivity is really delayed for the host.   This summarizes that we
do have problems to fix.

(b)    DS-Lite is an additional problem.   Since DS-Lite mandates that
the CPE and the AFTR perform fragmentation and reassembly, we have a
nasty problem.   Reassembly of tunneled encapsulated packets is very
complex because the receiver of the fragmented packet has to reassemble
before decapsulation.   Thus the received needs more memory and a
general purpose cpu.  If I want to choke a DS-Lite deployment at the
AFTR and the CPE, I will generate packets close to 1500 bytes and force
tunnel fragmentation.   Thus the DS-Lite problem is an attack vector
that Daniel Rosen pointed out too.

Further, PMTUD is not practical to deploy so the TCP MSS Adjust is still
a usable choice.    

Thanks,

Hemant

---

From: "Hemant Singh (shemant)" <[email-masked]>
Date: Mon, 14 May 2012 11:25:25 -0500

Chris and Victor,

Did you read this email before replying?  The issue is not just DS-Lite
specific and the problem is real for any of DS-Lite, 6rd, and native
IPv6/IPv4.  The network for native IPv6 can easily include one IPv6
router in the path that issues an ICMPv6 error with packet too big for
any packet size > 1280.   Now what does the native IPv6 cable deployment
do?   PMTUD is not a viable solution.   So how else are the problems
solved unless the TCP MSS Adjust is used at the BR, AFTR, or the CPE?
Additionally, the proposed text mandates that the TCP MSS Adjust feature
is turned off by default.

Further, what are cable operators saying for IPv4 CPE routers deployed
already that support the TCP MSS Adjust feature and the CPEs are
tweaking the value?  Cable, specifically does not use PPPOE or rarely
and thus issues for PPPOE and MTU are not even seen in cable deployment.
See the email from Randy Bush who says his home PPPOE deployment has
down-tuned.  

Regards,

Hemant

From: [email-masked] [mailto:[email-masked]] On Behalf
Of Hemant Singh (shemant)
Sent: Monday, May 14, 2012 11:13 AM
To: IPv6 Operations
Subject: [v6ops] proposed TCP MSS text for rfc6204bis

Thanks,

Hemant

---

From: "STARK, BARBARA H" <[email-masked]>
Date: Mon, 14 May 2012 16:48:46 +0000

CE Routers that do PPPoE should already know what to do around MTU. We don't need an additional requirement on this for their sake. I'm not aware of any new PPPoE-supporting CE routers that have trouble with MTU. Those days are long gone.

I don't think we need a requirement on this topic. Especially for a mandatory function that is off by default. Telling a user with a retail-purchased CE router to turn 6rd on if they want IPv6 is one thing. Telling them to look for some TCP MSS Adjust function and try turning it on if their connection occasionally seems to have problems is very different. This doesn't sound useful to me.
Barbara

From: [email-masked] [mailto:[email-masked]] On Behalf Of Hemant Singh (shemant)
Sent: Monday, May 14, 2012 12:25 PM
To: Hemant Singh (shemant); IPv6 Operations
Subject: Re: [v6ops] proposed TCP MSS text for rfc6204bis

Chris and Victor,

Did you read this email before replying?  The issue is not just DS-Lite specific and the problem is real for any of DS-Lite, 6rd, and native IPv6/IPv4.  The network for native IPv6 can easily include one IPv6 router in the path that issues an ICMPv6 error with packet too big for any packet size > 1280.   Now what does the native IPv6 cable deployment do?   PMTUD is not a viable solution.   So how else are the problems solved unless the TCP MSS Adjust is used at the BR, AFTR, or the CPE?  Additionally, the proposed text mandates that the TCP MSS Adjust feature is turned off by default.

Further, what are cable operators saying for IPv4 CPE routers deployed already that support the TCP MSS Adjust feature and the CPEs are tweaking the value?  Cable, specifically does not use PPPOE or rarely and thus issues for PPPOE and MTU are not even seen in cable deployment.  See the email from Randy Bush who says his home PPPOE deployment has down-tuned.

Regards,

Hemant

From: [email-masked]<mailto:[email-masked]> [mailto:[email-masked]]<mailto:[mailto:[email-masked]]> On Behalf Of Hemant Singh (shemant)
Sent: Monday, May 14, 2012 11:13 AM
To: IPv6 Operations
Subject: [v6ops] proposed TCP MSS text for rfc6204bis

[The CPE router MUST support a TCP MSS Adjust feature on packets traversing the CPE router.  By default, the TCP MSS Adjust feature is turned off. ]

The TCP MSS Adjust is not needed for TCP packets sourced or destined to the CPE because the number of such packets at the CPE will be very small.  Additionally, we (Wes and I) deliberately left out qualifying the feature for IPv4 or IPv6 because the feature should be supported for both since the CPE also supports 6rd.   We have left out specifying any default value for the MSS because there are several values since the MSS is a function of the protocols being used.   Note IPv4 CPE routers already support a TCP MSS Adjust knob.    Further, the DS-Lite AFTR and the 6rd BR in the SP domain sit between the home TCP client and a TCP server on the Internet.  Thus the AFTR and the BR can perform TCP MSS Adjust.  It's only for 6rd  where the CPE's talk without going through the BR, the CPE has to invoke the TCP MSS Adjust.

(a)    Any of native IPv6  and tunneled technologies such as DS-Lite and 6rd can cause ICMPv6 errors for packet too big to the source.   Even when the CPE issues the ICMPv6 error to the host connected to the CPE, the Internet access of the host is delayed which is not good.   Additionally, what if the CPE passes the host packet to the Internet and one router on the Internet issues the ICMPv6 error with packet too big but a node in the path back blocks the ICMPv6 error.   Now the Internet connectivity is really delayed for the host.   This summarizes that we do have problems to fix.

(b)   DS-Lite is an additional problem.   Since DS-Lite mandates that the CPE and the AFTR perform fragmentation and reassembly, we have a nasty problem.   Reassembly of tunneled encapsulated packets is very complex because the receiver of the fragmented packet has to reassemble before decapsulation.   Thus the received needs more memory and a general purpose cpu.  If I want to choke a DS-Lite deployment at the AFTR and the CPE, I will generate packets close to 1500 bytes and force tunnel fragmentation.   Thus the DS-Lite problem is an attack vector that Daniel Rosen pointed out too.

Further, PMTUD is not practical to deploy so the TCP MSS Adjust is still a usable choice.

Thanks,

Hemant

---

From: Chris Donley <[email-masked]>
Date: Mon, 14 May 2012 10:55:09 -0600

Hemant,

I'm not saying preclude MSS rewrite, I'm just saying don't mandate it.

Chris

From: Hemant Singh <[email-masked]<mailto:[email-masked]>>
Date: Mon, 14 May 2012 10:25:25 -0600
To: Hemant Singh <[email-masked]<mailto:[email-masked]>>, IPv6 Operations <[email-masked]<mailto:[email-masked]>>
Subject: Re: [v6ops] proposed TCP MSS text for rfc6204bis

Chris and Victor,

Regards,

Hemant

From: [email-masked]<mailto:[email-masked]> [mailto:[email-masked]] On Behalf Of Hemant Singh (shemant)
Sent: Monday, May 14, 2012 11:13 AM
To: IPv6 Operations
Subject: [v6ops] proposed TCP MSS text for rfc6204bis

The TCP MSS Adjust is not needed for TCP packets sourced or destined to the CPE because the number of such packets at the CPE will be very small.  Additionally, we (Wes and I) deliberately left out qualifying the feature for IPv4 or IPv6 because the feature should be supported for both since the CPE also supports 6rd.   We have left out specifying any default value for the MSS because there are several values since the MSS is a function of the protocols being used.   Note IPv4 CPE routers already support a TCP MSS Adjust knob.    Further, the DS-Lite AFTR and the 6rd BR in the SP domain sit between the home TCP client and a TCP server on the Internet.  Thus the AFTR and the BR can perform TCP MSS Adjust.  It’s only for 6rd  where the CPE’s talk without going through the BR, the CPE has to invoke the TCP MSS Adjust.

Thanks,

Hemant

---

From: "Hemant Singh (shemant)" <[email-masked]>
Date: Tue, 15 May 2012 10:06:55 -0500

Changed the TCP MSS Adjust from a MUST to a MAY.   In case one asks "why
have a MAY requirement at all?"  The issue makes sense to capture in
rfc6204bis so that the community is aware of potential problems and
especially since RFC 4459 does not discuss the TCP MSS Adjust.  Changed
text is included below.  Latest by tomorrow, the -09 for rfc6204bis is
planned to be released. 

OLD

4.4.  Transition Technologies Support

4.4.1.  6rd

NEW

4.4.  Transition Technologies Support
 
   Transition technology support requirements:
 
   TTS-1:  The CPE router MAY support a TCP MSS Adjust feature on
           packets traversing the CPE router.
 
4.4.1.  6rd

Hemant

Thanks,

Hemant

---

From: Ole Trøan <[email-masked]>
Date: Tue, 15 May 2012 17:12:08 +0200

> Changed the TCP MSS Adjust from a MUST to a MAY.   In case one asks “why have a MAY requirement at all?”  The issue makes sense to capture in rfc6204bis so that the community is aware of potential problems and especially since RFC 4459 does not discuss the TCP MSS Adjust.  Changed text is included below.  Latest by tomorrow, the -09 for rfc6204bis is planned to be released.
>  
> OLD
>  

I object to any TCP MSS text in the document.

cheers,
Ole

---

From: Ole Trøan <[email-masked]>
Date: Tue, 15 May 2012 18:33:41 +0200

> I object to any TCP MSS text in the document.

I got poked to give a reason for this opinion:

top of my head:
- don't want an IPv6 router to have to do this forever
- don't want to require all routers to look into transport layer information.
  (aka layer violation)
- we never specified this for IPv4
- is TCP MSS rewrite specified anywhere to reference? or are we now going to specify that in an v6ops
  document?
- this isn't specific to transition mechanisms
  and isn't such an issue given that the MTU for 6rd, DS-lite are required to be "well managed"
- doesn't solve the problem.
  only works for TCP. only handles the case of a WAN-side MTU being lower than LAN side MTU
  use RFC481 for TCP
- should be reviewed by the transport area

cheers,
Ole

---

From: Marc Blanchet <[email-masked]>
Date: Tue, 15 May 2012 12:46:03 -0400

> - is TCP MSS rewrite specified anywhere to reference? or are we now going to specify that in an v6ops

I agree with Ole that MSS rewrite is not specific to this document and requires additional detailed specification on how it is done. 
So I'm also opposed in adding this requirement as it is now (i.e. underspecified and not discussing all the consequences).

So either 
a) we identify a document that clearly describes how to do MSS rewrite, the pros and cons, the applicability, etc… and then we refer to that document. Does such document exist?
b) we do not specify MSS rewrite here.

Regards, Marc.

> _______________________________________________
> v6ops mailing list
> [email-masked]
> https://www.ietf.org/mailman/listinfo/v6ops

---

From: "Templin, Fred L" <[email-masked]>
Date: Tue, 15 May 2012 09:53:11 -0700

If the concern is for MTU mitigations for tunneling protocols
that use IPv4 as the outer layer of encapsulation, SEAL has a
solution for this:

https://datatracker.ietf.org/doc/draft-templin-intarea-seal/

The idea is that the tunneling protocol has an adjunct UDP
messaging protocol whereby the tunnel egress informs the
ingress in case any IPv4 fragmentation occurs on the path.
The MTU feedback is delivered in the form of a "Packet Too
Big" message that the ingress can then turn around and
return to the original source.

So, I agree with Ole - no need to have MSS adjustments if
we can make path MTU discovery more robust.

Fred
[email-masked]

> -----Original Message-----
> From: [email-masked] [mailto:[email-masked]] On Behalf Of
> Ole Trøan
> Sent: Tuesday, May 15, 2012 9:34 AM
> To: Hemant Singh
> Cc: IPv6 Operations

---

From: Cameron Byrne <[email-masked]>
Date: Tue, 15 May 2012 09:56:40 -0700

>>> I object to any TCP MSS text in the document.

I object to your objection !

> b) we do not specify MSS rewrite here.

c). 6204bis CE MAY implement a compensating mechanism for MTU related
service degradation that may occur when tunnels such as 6RD and
DS-lite are used.

CB

---

From: Rémi Després <[email-masked]>
Date: Tue, 15 May 2012 19:05:24 +0200

Le 2012-05-15 à 18:56, Cameron Byrne a écrit :

> DS-lite are used

... in particular by some ad hoc MSS reductions in TCP SYN packets [RFC879]

c) is preferred to doing nothing especially if completed as proposed here.

Free, the largest 6rd network, does AFAIK MSS rewrites (copy added to Alexandre who can confirm or infirm).

RD

---

From: Warren Kumari <[email-masked]>
Date: Tue, 15 May 2012 13:29:38 -0400

> - should be reviewed by the transport area

Having used widgets that did MSS rewriting (to make IPv4 TCP packets fit down a GRE tunnel, that itself rode on IPSec) I ran into (at least) these problems:
1: The widget would occasionally rewrite what it though was the MSS field in *non-SYN* packets. This obviously does not work (and is, er, fun to debug).
2: The CPU load was much higher with the MSS rewrite feature on. The widget had hardware that did session state tracking, and the MSS rewrite function required punting from the fast path to the CPU. In another widget (actually just a newer version of the same widget) the MSS rewrite was done in fast path, but lots of SYN packets would overwhelm the devices ability to rewrite (and at a much lower rate than with rewriting turned off) -- as with many hardware limits, the fact that you had bumped into it wasn't exposed anywhere.
3: With a different vendor -- If you happened to try run a (GRE) tunnel through the device, it would "helpfully" peer into the tunnel and rewrite the MSS on the tunnels SYN packet, correctly update the TCP checksum, but then fail to recalculate the GRE checksum. This "feature" seemed to not be something you could turn off...

Yes, all of these are anecdotes, and yes, all of them were stupid implementation bugs, but MSS rewrite caused much annoyance...

W

---

From: james woodyatt <[email-masked]>
Date: Tue, 15 May 2012 10:59:40 -0700

>> I object to any TCP MSS text in the document.

I share Mr. Troan's opinion, and I specifically object to the MAY clause quoted above on the grounds that it conveys the rather undesirable idea that IETF believes it is truly OPTIONAL whether routers will or will not rewrite TCP headers in forwarded packets.  I see no good reason to promote this debauchery by affirmatively declaring tolerance of it.

--
james woodyatt <[email-masked]>
member of technical staff, core os networking

---

From: "Templin, Fred L" <[email-masked]>
Date: Tue, 15 May 2012 11:04:26 -0700

Hi Remi,

> ... in particular by some ad hoc MSS reductions in TCP SYN packets

That's a great RFC, but I didn't see anything about
gateways rewriting the MSS (or maybe that's not what
you were intending to say).

> Alexandre who can confirm or infirm).

I have a different idea for 6rd MTU mitigations:

1) Set the 6rd tunnel MTU to 64KB
2) Set DF=0 in the outer IPv4 header of each packet
   admitted into the tunnel
3) The 6rd egress tunnel endpoint watches for any
   tunneled packets that arrive fragmented. When a
   fragmented tunneled packet arrives, the 6rd egress
   tunnel endpoint drops it and sends a UDP message
   to the 6rd tunnel ingress saying: "Packet Too Big".
4) The 6rd ingress tunnel endpoint receives the
   UDP PTB message, translates it into an ICMPv6
   PTB message, and returns the message to the
   original IPv6 source within the 6rd site.

Thanks - Fred

---

From: "Weil, Jason" <[email-masked]>
Date: Tue, 15 May 2012 14:12:44 -0400

+1

..and back to an earlier discussion, the MSS rewrite can but does not need
to be done on the CPE Router. As a provider, If I chose to deploy an AFTR
to support DS-Lite I would make sure the AFTR did the MSS rewrite since it
is the one box I actually administer. I would want the service to be as
controllable as possible so that I could tune as needed. A MAY here is not
needed and should be dropped imo.

Jason

>- should be reviewed by the transport area

This E-mail and any of its attachments may contain Time Warner Cable proprietary information, which is privileged, confidential, or subject to copyright belonging to Time Warner Cable. This E-mail is intended solely for the use of the individual or entity to which it is addressed. If you are not the intended recipient of this E-mail, you are hereby notified that any dissemination, distribution, copying, or action taken in relation to the contents of and attachments to this E-mail is strictly prohibited and may be unlawful. If you have received this E-mail in error, please notify the sender immediately and permanently delete the original and any copy of this E-mail and any printout.

---

From: Alexandre Cassen <[email-masked]>
Date: Tue, 15 May 2012 21:02:02 +0200 (CEST)

> Alexandre who can confirm or infirm).

not for 6rd, we do not perform mss_adjust at 6rd tuneling. In other 
tuneling playground we are using mss_adjust, yes, but not here.

---

From: "Hemant Singh (shemant)" <[email-masked]>
Date: Tue, 15 May 2012 14:25:03 -0500

-----Original Message-----
From: Templin, Fred L [mailto:[email-masked]] 
Sent: Tuesday, May 15, 2012 12:53 PM
To: Ole Trøan; Hemant Singh (shemant)
Cc: IPv6 Operations
Subject: RE: [v6ops] proposed TCP MSS text for rfc6204bis

>we can make path MTU discovery more robust.

PMTUD does not work across the Internet as already discussed related to this thread; a node in the path may block ICMPv6 error to be propagated back to the source of the packet too big.  Also said was that even if the ICMPv6 error makes it to the host, the host delays Internet connectivity for the user because the host processes the ICMPv6 error and works out out packet fragmentation.  For a tunneled tech such as DS-Lite the host the CPE is mandated to do tunnel fragmentation.  This is an interesting conundrum because DS-Lite did not ask the host to fragment the payload but the host does fragment the payload.  This issue is resolved if the CPE implements the TCP MSS Adjust.

Additionally, it has be pointed out in earlier emails that when 6rd CPEs communicate directly to each other without going thru the BR, then the CPE is a good node in the path between the TCP server and the TCP client to perform the TCP MSS Adjust.

Hemant

---

From: "Hemant Singh (shemant)" <[email-masked]>
Date: Tue, 15 May 2012 14:32:22 -0500

Jason,

There is no IETF document including the DS-Lite RFC 6333 that calls out for MSS Rewrite at the DS-Lite AFTR.  We also have an RFC in RFC 4459 that covers fragmentation for tunneled technologies but the RFC does not cover TCP MSS Adjust.  That is why some folks have asked for the issue to be captured in rfc6204bis and also because there are clear use cases where the CPE is best doing the TCP MSS Adjust such as when 6rd nodes are communicating directly with each other.

A simple case when 6rd CPEs talk directly to each other.   A host has MTU default of 1500.  The CPE uses PPPOE in 6rd and PPPOE has MTU of 1492.  The host sent a packet of size 1500.  The CPE drops the packet and issues an ICMPv6 error of packet too big.  Now the host has to deal with fragmentation and Internet connectivity of the host is delayed.

Hemant

-----Original Message-----
From: Weil, Jason [mailto:[email-masked]] 
Sent: Tuesday, May 15, 2012 2:13 PM
To: Ole Trøan; Hemant Singh (shemant)
Cc: IPv6 Operations
Subject: Re: [v6ops] proposed TCP MSS text for rfc6204bis

+1

Jason

---

From: Simon Perreault <[email-masked]>
Date: Tue, 15 May 2012 15:32:49 -0400

>                       packets traversing the CPE router.

-1

The statement is too general. Allowing MSS rewriting all the time on all 
packets under any circumstance is wrong. It has to be qualified carefully.

Either we spend time carefully qualifying the statement, or we leave it 
as future work for a separate RFC. I vote for the latter.

Simon
-- 
DTN made easy, lean, and smart --> http://postellation.viagenie.ca
NAT64/DNS64 open-source        --> http://ecdysis.viagenie.ca
STUN/TURN server               --> http://numb.viagenie.ca

---

From: Ole Trøan <[email-masked]>
Date: Tue, 15 May 2012 21:39:32 +0200

> A simple case when 6rd CPEs talk directly to each other.   A host has MTU default of 1500.  The CPE uses PPPOE in 6rd and PPPOE has MTU of 1492.  The host sent a packet of size 1500.  The CPE drops…

please read section 9.1 of RFC5969.

Ole

---

From: "Templin, Fred L" <[email-masked]>
Date: Tue, 15 May 2012 13:29:58 -0700

Hi Hemant,

> PMTUD does not work across the Internet as already discussed
> related to this thread; a node in the path may block ICMPv6
> error to be propagated back to the source of the packet too
> big.

Let's examine the components of an IPv6 path supported
by a 6rd tunnel:

  1) the path within the 6rd site between the IPv6
     host and the 6rd CPE.
  2) the path within the operator's network between
     the CPE and the 6rd BR.
  3) the path from the BR across the IPv6 Internet to
     the final destination's site.
  4) the path from the final destination's site border
     router to the final destination.

For component 1), IPv6 PMTUD must surely work in
order for heterogeneous link MTUs to be active
within the site.

For components 3) and 4), IPv6 PMTUD must surely
work in order for the original source to obtain
useful PMTUD feedback *even if some router is
adjusting the MSS*. This is because the links
within the component 3) and 4) paths may have a
too-small MTU to support the MSS chosen by the
router. 

For component 2), it is not possible to rely on
IPv4 PMTUD due to the many well-known issues.
That is why my proposal is to disable IPv4 PMTUD
by setting DF=0 and having the 6rd tunnel egress
return the IPv6 PMTUD feedback (and, I just realized
that UDP encaps is unnecessary because the 6rd
egress can simply use plain IPv6/IPv4 encapsulation
to return the ICMPv6 PTB message back to the ingress).

> Also said was that even if the ICMPv6 error makes it
> to the host, the host delays Internet connectivity for the
> user because the host processes the ICMPv6 error and works
> out out packet fragmentation.  For a tunneled tech such as
> DS-Lite the host the CPE is mandated to do tunnel fragmentation.
> This is an interesting conundrum because DS-Lite did not ask
> the host to fragment the payload but the host does fragment
> the payload.  This issue is resolved if the CPE implements
> the TCP MSS Adjust.

I find this all to be very weak reasoning. IF the
host gets the necessary PMTUD feedback it will do
the right thing - it's been that way for over 20
years. Any tweaks by middleboxes that try to give
small optimizations to hosts are just best-guesses
as to how the hosts will react.

> Additionally, it has be pointed out in earlier emails that
> when 6rd CPEs communicate directly to each other without
> going thru the BR, then the CPE is a good node in the path
> between the TCP server and the TCP client to perform the
> TCP MSS Adjust.

For the CPE-to-CPE path, it works the just the same
way as for the CPE-to-BR path. In neither case does
a CPE have to adjust MSS if MTU discovery is
properly implemented.

> Hemant

Fred (and please do not turn this back into RTF)

---

From: "Hemant Singh (shemant)" <[email-masked]>
Date: Tue, 15 May 2012 16:58:19 -0500

-----Original Message-----
From: Ole Troan [mailto:[email-masked]] On Behalf Of Ole Trøan
Sent: Tuesday, May 15, 2012 3:40 PM
To: Hemant Singh (shemant)
Cc: Weil, Jason; IPv6 Operations
Subject: Re: [v6ops] proposed TCP MSS text for rfc6204bis

>please read section 9.1 of RFC5969.

The section includes recommendations on the 6rd tunnel MTU.  However, the deployment cannot control that the default MTU on most popular host operating systems is 1500.  Thus if the CPE is set to anything smaller than 1500 for an MTU, the use case of host packet drop at the CPE and the CPE issuing ICMPv6 Error for packet too big to the host stands.  Then one has UnHappyEyeballs.

TCP MSS Adjust at the CPE fixes the issue by adjusting the MSS so that the host TCP client uses the MSS value. 

Hemant

---

From: "Templin, Fred L" <[email-masked]>
Date: Tue, 15 May 2012 15:28:29 -0700

> The section includes recommendations on the 6rd tunnel
> MTU. However, the deployment cannot control that the
> default MTU on most popular host operating systems is
> 1500.  Thus if the CPE is set to anything smaller than
> 1500 for an MTU, the use case of host packet drop at
> the CPE and the CPE issuing ICMPv6 Error for packet
> too big to the host stands.  Then one has UnHappyEyeballs.

Why? IPv6 PMTUD within the site works. For that
matter, IPv6 PMTUD over any all-IPv6 path works.
If that were not to be the case, then no choice
but to set MTU=1280 on all links.

We should only be worried about the MTU of any
portion of the path that is manifested by IPv6
in IPv4 tunneling, i.e. the 6rd tunnel. My proposal 
fixes that portion of the path.

> TCP MSS Adjust at the CPE fixes the issue by adjusting
> the MSS so that the host TCP client uses the MSS value.

If the CPE adjusts MSS to anything larger than 1240,
there may still be some other link on the path to the
final destination that configures a too-small MTU to
support the chosen MSS. In that case, an ICMPv6 PTB
is generated and you still have to rely on IPv6 PMTUD.
And, adjusting MSS to 1240 would (IMHO) be a very
unsatisfactory outcome.

Fred

---

From: Hesham Soliman <[email-masked]>
Date: Wed, 16 May 2012 12:04:54 +1000

>
>I agree with Ole that MSS rewrite is not specific to this document and
>requires additional detailed specification on how it is done.
>So I'm also opposed in adding this requirement as it is now (i.e.
>underspecified and not discussing all the consequences).
>
>So either 
>a) we identify a document that clearly describes how to do MSS rewrite,
>the pros and cons, the applicability, etcŠ and then we refer to that
>document. Does such document exist?

=> +1. I would like to add that such document should be reviewed by the
Transport area. 

Hesham

---

From: Lorenzo Colitti <[email-masked]>
Date: Wed, 16 May 2012 12:37:35 +0900

On Tue, May 15, 2012 at 12:13 AM, Hemant Singh (shemant)
<[email-masked]>wrote:

> (a)    **Any of *native IPv6*  and tunneled technologies such as DS-Lite
> and 6rd can cause ICMPv6 errors for packet too big to the source.   Even
> when the CPE issues the ICMPv6 error to the host connected to the CPE, the
> Internet access of the host is delayed which is not good.   Additionally,
> what if the CPE passes the host packet to the Internet and one router on
> the Internet issues the ICMPv6 error with packet too big but a node in the
> path back blocks the ICMPv6 error.   Now the Internet connectivity is
> really delayed for the host.   This summarizes that we do have problems to
> fix.
>

I strongly disagree that this is a CPE problem. The node in the path is
doing it wrong and needs to be fixed. The CPE should not work around this
problem.

---

From: Wuyts Carl <[email-masked]>
Date: Wed, 16 May 2012 07:57:25 +0200

+1, PMTUD it should be!! No MSS.  Please tackle the problem where it pops up, i.e. if certain nodes don't do their jobs on ICMPv6 handling, they're not qualified for the role, so it should be fixed at those places.  One keeps trying to push some "improvements" to the CPEs while it's not the CPE implementing anything wrong.  
Don't get me wrong, I don't ignore the issue being present.  We've been blamed already before that our PPP was not working OK for IPV6, but in the end, some node was dropping the ICMPv6 messages, hence dropped packets due to MTU size.

regs
Carl

-----Original Message-----
From: [email-masked] [mailto:[email-masked]] On Behalf Of Templin, Fred L
Sent: dinsdag 15 mei 2012 22:30
To: Hemant Singh (shemant); Ole Trøan
Cc: IPv6 Operations
Subject: Re: [v6ops] proposed TCP MSS text for rfc6204bis

Hi Hemant,

>> So, I agree with Ole - no need to have MSS adjustments if we can make 
>> path MTU discovery more robust.
>
> PMTUD does not work across the Internet as already discussed related 
> to this thread; a node in the path may block ICMPv6 error to be 
> propagated back to the source of the packet too big.

Let's examine the components of an IPv6 path supported by a 6rd tunnel:

For component 1), IPv6 PMTUD must surely work in order for heterogeneous link MTUs to be active within the site.

For components 3) and 4), IPv6 PMTUD must surely work in order for the original source to obtain useful PMTUD feedback *even if some router is adjusting the MSS*. This is because the links within the component 3) and 4) paths may have a too-small MTU to support the MSS chosen by the router. 

For component 2), it is not possible to rely on
IPv4 PMTUD due to the many well-known issues.
That is why my proposal is to disable IPv4 PMTUD by setting DF=0 and having the 6rd tunnel egress return the IPv6 PMTUD feedback (and, I just realized that UDP encaps is unnecessary because the 6rd egress can simply use plain IPv6/IPv4 encapsulation to return the ICMPv6 PTB message back to the ingress).

> Also said was that even if the ICMPv6 error makes it to the host, the 
> host delays Internet connectivity for the user because the host 
> processes the ICMPv6 error and works out out packet fragmentation.  
> For a tunneled tech such as DS-Lite the host the CPE is mandated to do 
> tunnel fragmentation.
> This is an interesting conundrum because DS-Lite did not ask the host 
> to fragment the payload but the host does fragment the payload.  This 
> issue is resolved if the CPE implements the TCP MSS Adjust.

I find this all to be very weak reasoning. IF the host gets the necessary PMTUD feedback it will do the right thing - it's been that way for over 20 years. Any tweaks by middleboxes that try to give small optimizations to hosts are just best-guesses as to how the hosts will react.

> Additionally, it has be pointed out in earlier emails that when 6rd 
> CPEs communicate directly to each other without going thru the BR, 
> then the CPE is a good node in the path between the TCP server and the 
> TCP client to perform the TCP MSS Adjust.

For the CPE-to-CPE path, it works the just the same way as for the CPE-to-BR path. In neither case does a CPE have to adjust MSS if MTU discovery is properly implemented.

> Hemant

Fred (and please do not turn this back into RTF) _______________________________________________
v6ops mailing list
[email-masked]
https://www.ietf.org/mailman/listinfo/v6ops

---

From: Lorenzo Colitti <[email-masked]>
Date: Wed, 16 May 2012 15:59:35 +0900

On Wed, May 16, 2012 at 2:57 PM, Wuyts Carl <[email-masked]>wrote:

> +1, PMTUD it should be!! No MSS.  Please tackle the problem where it pops
> up, i.e. if certain nodes don't do their jobs on ICMPv6 handling, they're
> not qualified for the role, so it should be fixed at those places.

Even if you fix all those places, PMTUD still adds one RTT to every
connection you make to a new host. Do we want to live with that kind of
performance penalty compared to IPv4 - or, are you willing to have strict
happy eyeballs implementation always prefer IPv4 because IPv6 is slow to
establish connections

I'm not saying the solution should be MSS rewriting - I don't think it
should be. But if you're a 6rd CPE and you know your MTU to the Internet is
1480, then giving the host an MTU of 1480 in the RA would improve TCP
latency.

---

From: Wuyts Carl <[email-masked]>
Date: Wed, 16 May 2012 09:02:55 +0200

@ Lorenzo
Our RTADVD allows for setting MTU (configurable), so can be applied for 6rd without any problems, so this has been taken into account already.  Just want to avoid "dirty" work-around in CPE-land with MSS-stuff.

Regs
Carl

From: Lorenzo Colitti [mailto:[email-masked]]
Sent: woensdag 16 mei 2012 9:00
To: Wuyts Carl
Cc: Templin, Fred L; Hemant Singh (shemant); Ole Trøan; IPv6 Operations
Subject: Re: [v6ops] proposed TCP MSS text for rfc6204bis

On Wed, May 16, 2012 at 2:57 PM, Wuyts Carl <[email-masked]<mailto:[email-masked]>> wrote:
+1, PMTUD it should be!! No MSS.  Please tackle the problem where it pops up, i.e. if certain nodes don't do their jobs on ICMPv6 handling, they're not qualified for the role, so it should be fixed at those places.

Even if you fix all those places, PMTUD still adds one RTT to every connection you make to a new host. Do we want to live with that kind of performance penalty compared to IPv4 - or, are you willing to have strict happy eyeballs implementation always prefer IPv4 because IPv6 is slow to establish connections

I'm not saying the solution should be MSS rewriting - I don't think it should be. But if you're a 6rd CPE and you know your MTU to the Internet is 1480, then giving the host an MTU of 1480 in the RA would improve TCP latency.

---

From: Rémi Després <[email-masked]>
Date: Wed, 16 May 2012 09:08:56 +0200

Le 2012-05-15 à 21:02, Alexandre Cassen a écrit :

> not for 6rd, we do not perform mss_adjust at 6rd tuneling. In other tuneling playground we are using mss_adjust, yes, but not here.

I suppose it is acceptable because 1280, the common IPv6 transmit MTU, purposely chosen well below the 1500 common link MTU, the need is not the same as for some other tunnels.
  
Thanks for the clarification.

RD

---

From: Rémi Després <[email-masked]>
Date: Wed, 16 May 2012 09:13:23 +0200

Hi Fred,

> you were intending to say).

This RFC, which is about CE routers, isn't concerned with what other gateways might do.

Regards,
RD

---

From: Rémi Després <[email-masked]>
Date: Wed, 16 May 2012 09:25:13 +0200

2012-05-15 à 19:59, james woodyatt:

> I share Mr. Troan's opinion, and I specifically object to the MAY clause quoted above on the grounds that it conveys the rather undesirable idea that IETF believes it is truly OPTIONAL whether routers will or will not rewrite TCP headers in forwarded packets.  

> I see no good reason to promote this debauchery by affirmatively declaring tolerance of it.

For clarification of why you say this:
- Is it that you negate that MSS rewrite can prevent adverse effects of fragmentation where TCP ICMP-PTB packets are lost or ignored?
- Or is it because such prevention uses some inter-layer cooperation (very limited but real)?

Regards,
RD

---

From: Lorenzo Colitti <[email-masked]>
Date: Wed, 16 May 2012 16:36:42 +0900

> - Or is it because such prevention uses some inter-layer cooperation (very

I think what he said is:

- It's superfluous
- It's a layering violation
- The IETF should not endorse it by making it optional

I would like to add that superfluous -> slow.

---

From: Mark Andrews <[email-masked]>
Date: Wed, 16 May 2012 18:07:17 +1000

In message <CAKD1Yr2vAxzwAk3MTr03_=[email-masked]>
> happy eyeballs implementation always prefer IPv4 because IPv6 is slow to

Happy Eyeballs doesn't see the PMTUD issues, they come later in the
transaction.  If you want Happy Eyeballs to see PMTUD then we need
a TCP option that pads the SYN|ACK to the computed MSS and does it
for both IPv4 and IPv6.  Happy Eyeballs will then prefer IPv6 if
DS-Lite is in use and IPv4 if 6to4 / 6rd are in use and be agnostic
if both are native.

-- 
Mark Andrews, ISC
1 Seymour St., Dundas Valley, NSW 2117, Australia
PHONE: +61 2 9871 4742                 INTERNET: [email-masked]

---

From: Rémi Després <[email-masked]>
Date: Wed, 16 May 2012 11:19:34 +0200

2012-05-16 à 09:36, Lorenzo Colitti:

> On Wed, May 16, 2012 at 4:25 PM, Rémi Després <[email-masked]> wrote:

Not vital, I agree, but not superfluous as long as ICMP PTB handling is known to be abundantly mistreated, and fragmentation can deteriorate performance.

> - It's a layering violation

Well, one can see it as a layer "cooperation"  (serves a useful purpose).

> - The IETF should not endorse it by making it optional

I can't understand why IETF wouldn't document, and permit as opposed to require, a solution that is known to do some good and no harm, and that has been successfully used in the field.

> I would like to add that superfluous -> slow.

Not sure to understand what you mean.

RD

---

From: Gert Doering <[email-masked]>
Date: Wed, 16 May 2012 11:24:40 +0200

Hi,

> happy eyeballs implementation always prefer IPv4 because IPv6 is slow to

I don't think HE will actually notice that delay, as it happens after
SYN/ACK handshaking - and HE is happy as soon as the connect() returns.

But indeed, this is not particularily nice, as the MTU-impaired link will
usually be close to the client, and the server's MTU needs to be adjusted.

OTOH, one might hope that eventually people will learn how to build 
proper ISP backbones that can handle a layer of ISP-network tunneling
(like "PPPoE") without MTU reduction...

Or one could advise the servers to assume 1492 byte MTU for the generic
eyeball case...

> 1480, then giving the host an MTU of 1480 in the RA would improve TCP

True (assuming that the incoming MTU is then also 1480 and that the host
would signal a proper MSS to the server).

Outbound packets from the client wouldn't suffer that much, as only the
LAN RTT will be added.

Gert Doering
        -- NetMaster
-- 
have you enabled IPv6 on something today...?

SpaceNet AG                        Vorstand: Sebastian v. Bomhard
Joseph-Dollinger-Bogen 14          Aufsichtsratsvors.: A. Grundner-Culemann
D-80807 Muenchen                   HRB: 136055 (AG Muenchen)
Tel: +49 (89) 32356-444            USt-IdNr.: DE813185279

---

From: Daniel Roesen <[email-masked]>
Date: Wed, 16 May 2012 12:55:22 +0200

> (like "PPPoE") without MTU reduction...

Sometimes it's not that easy. E.g. how do you manage endpoint-specific
MTU in a shared L3 broadcast domain? There is a proposal for an IPv6 NDP
extension to negotiate bilateral MTU, but nothing standardised, let
alone generally implemented.

Best regards,
Daniel

-- 
CLUE-RIPE -- Jabber: [email-masked] -- dr@IRCnet -- PGP: 0xA85C8AA0

---

From: [email-masked]
Date: Wed, 16 May 2012 08:37:24 -0400

>> +1, PMTUD it should be!! No MSS.  
should 
> be. But if you're a 6rd CPE and you know your MTU to the Internet is 
1480, 
> then giving the host an MTU of 1480 in the RA would improve TCP latency.

+1 
I second Lorenzo on this (no MSS req, PMTUD often doesn't work or is slow)

MSS is messy and PMTUD doesn't work in many cases (by experience for 6RD). 

The only workable solution in the short term IMHO is reducing the LAN MTU
in the RA. Several CPEs already do it, with success (the number of MTU 
issues is reduced to almost 0). 

/JF

---

From: "Templin, Fred L" <[email-masked]>
Date: Wed, 16 May 2012 08:52:30 -0700

Hi,

I think we should get something clear here. PMTUD for
IPv6 *works*. Were that not to be the case, then we
are stuck with 1500 or smaller MTUs *forever*.

6rd is just one IPv6 hop in the path, and must support
IPv6 PMTUD the same as for any IPv6 hop. In its current
form, 6rd has difficulty with any MTU setting greater
than 1480, and may even have problems at that level.
So, PMTUD for 6rd should be fixed, which is exactly
what my proposal does.

Thanks - Fred

---

From: "Templin, Fred L" <[email-masked]>
Date: Wed, 16 May 2012 09:17:58 -0700

Forgot to say, there is also RFC4821. So, even if there
are doubts about IPv6 PMTUD the hosts still have recourse
to police the PMTU themselves. But again, that is an IPv6
*path* consideration and not a 6rd consideration.

Fred

> Templin, Fred L
> Sent: Wednesday, May 16, 2012 8:53 AM
> To: IPv6 Operations

---

From: "Templin, Fred L" <[email-masked]>
Date: Wed, 16 May 2012 09:46:28 -0700

Forgot also to say that RFC4821 takes care of the RTT issue,
since the data is delivered w/o a RTT delay even if there is
an MTU restriction on the path.

Fred

> Sent: Wednesday, May 16, 2012 9:18 AM
> To: IPv6 Operations
> Of

---

From: Warren Kumari <[email-masked]>
Date: Wed, 16 May 2012 16:39:19 -0400

> I would like to add that superfluous -> slow.

And I'll add another -- it's a truly horrendously ugly kludge.

Intermediate devices monkeying with my packet -- do not want….

W

P.S: Also, while I'm on my soapbox: This thread becoming RTF again -- do not want...

---

From: Randy Bush <[email-masked]>
Date: Wed, 16 May 2012 10:54:31 -1000

> +1, PMTUD it should be!

and cash should fall from the sky

let me reposition the issue

whether we like it or not, many of these hacks, and hacks they are, will
break.  pmtud would be lovely, but no prudent op can bet on it.
therefore, we need to give
  o warning(s)
  o some guidance

in what document(s) and how are those best done?

randy

---

From: "Hemant Singh (shemant)" <[email-masked]>
Date: Wed, 16 May 2012 18:25:58 -0500

-----Original Message-----
From: [email-masked] [mailto:[email-masked]] On Behalf
Of Warren Kumari
Sent: Wednesday, May 16, 2012 4:39 PM
To: Lorenzo Colitti
Cc: [email-masked] WG
Subject: Re: [v6ops] proposed TCP MSS text for rfc6204bis

>Intermediate devices monkeying with my packet -- do not want....

For over a decade the middlebox edited the TCP MSS for IPv4 and saved
one's PPPOE Internet connection from getting dropped.  Hmm, which is
worse?  No Internet connection vs. the wish above.

PMTUD is not an option since the protocol uses ICMPv6 that can be
blocked in the Internet path.  Fred Templin has ideas along the right
track but none of those ideas are in a Standards Track RFC that
rfc6204bis can reference for a technology.  Don't know how many hosts
implement RFC 4821 and PLPMTUD.  That leaves only two options at the
table. 

(a) The SP knows exactly what MTU to sent in an RA to the CPE WAN.  The
MTU is the effective MTU for DS-Lite and the MTU can be used in the RA
on the CPE LAN segment.  Daniel Rosen and Gert dislike the WAN MTU to
mess with the LAN MTU - if they can clarify why, that would be helpful.
Anyway, 6rd does not have an RA from the SP to the CPE WAN.  So the CPE
has some internal magic that if the CPE uses 6rd, the CPE uses a certain
conservative MTU such as 1280 on the CPE LAN or the CPE has a manual
configuration which is not good.  If Daniel and Gert find important
reasons then (a) is not a good option.  Note for DS-Lite the SP has
another alternative in that the SP sets the TCP MSS at the AFTR.

(b) Use TCP MSS Rewrite in the CPE because there are certain cases (6rd
CPE to CPE communication) where the SP cannot set the TCP MSS and hosts
in the CPE LAN send packets larger than the protocol MTU used in the
Internet connection from the CPE WAN to the SP.  Who signals to the
hosts to reduce MTU?  

Hemant

---

From: Mark Andrews <[email-masked]>
Date: Thu, 17 May 2012 09:38:42 +1000

> in what document(s) and how are those best done?

You got Comcast, ATT, etc. to look at whom their customers are
talking to and to configure a low MTU link and try to talk to those
boxes through it.  If they break then you report it.  Having 20+
major ISPs all complain at the same time that PMTUD is broken may
well have a positive effect.

Issue a CVE against all IPv6 boxes that don't implement RFC 4821.
RFC 4821 support should be manditory for IPv6.  The one exception
permitted is boxes that fragment *all* IPv6 packets at 1280.

This really is at the level where a CVE needs to be issued.

RFC 4821 is already mentioned in node requirements (RFC 6434) though
I think we could make the wording stronger.

Mark

---

From: Randy Bush <[email-masked]>
Date: Wed, 16 May 2012 13:53:15 -1000

> I think we could make the wording stronger.

amazingly us-centric for an aussie.  and thank you for the example of
waiting for cash to fall from the sku.

randy

---

From: "Templin, Fred L" <[email-masked]>
Date: Wed, 16 May 2012 17:00:17 -0700

>> Intermediate devices monkeying with my packet -- do not want..
>
> For over a decade the middlebox edited the TCP MSS for IPv4
> and saved one's PPPOE Internet connection from getting dropped.
> Hmm, which is worse?  No Internet connection vs. the wish above.
>
> PMTUD is not an option since the protocol uses ICMPv6 that can
> be blocked in the Internet path.

PMTUD is mandatory in IPv6 unless the minimum MTU of
1280 is used. The 6rd ISP has no way of controlling the
IPv6 PMTU "in the wild" beyond its own boundaries, so
there is no way for it to avoid dependence on ICMPv6
PMTUD even if the CPE router rewrites the MSS. (Unless
the CPE rewrites the MSS to a "silly" value such as
1240 or smaller).

> Fred Templin has ideas along the right track but none
> of those ideas are in a Standards Track RFC that rfc6204bis
> can reference for a technology.

It should not be a problem to put them into a
standards-track document especially given customer
pull such as from rfc6204bis.

> Don't know how many hosts implement RFC 4821 and PLPMTUD.

I believe at least one major OS vendor implements RFC4821.
Like Mark Andrews said, the recommendation for implementing
RFC4821 should be made as strong as possible.

> That leaves only two options at the table. 

> (a) The SP knows exactly what MTU to sent in an RA to the 
> CPE WAN.  The MTU is the effective MTU for DS-Lite and the 
> MTU can be used in the RA on the CPE LAN segment.  Daniel
> Rosen and Gert dislike the WAN MTU to mess with the LAN
> MTU - if they can clarify why, that would be helpful.
> Anyway, 6rd does not have an RA from the SP to the CPE
> WAN.  So the CPE has some internal magic that if the CPE
> uses 6rd, the CPE uses a certain conservative MTU such as
> 1280 on the CPE LAN or the CPE has a manual configuration
> which is not good.  If Daniel and Gert find important
> reasons then (a) is not a good option.  Note for DS-Lite
> the SP has another alternative in that the SP sets the
> TCP MSS at the AFTR.

> (b) Use TCP MSS Rewrite in the CPE because there are certain
> cases (6rd CPE to CPE communication) where the SP cannot set
> the TCP MSS and hosts in the CPE LAN send packets larger than
> the protocol MTU used in the Internet connection from the CPE
> WAN to the SP.  Who signals to the hosts to reduce MTU?

If the CPE rewrites TCP MSS to something larger than
the MTU of the smallest link on the path to the final
destination (minus 40 bytes), then ICMPv6 PMTUD will
ensue. If TCP MSS rewrite alone is the only mechanism
used to avoid PMTUD issues, then the only safe rewrite
value would by 1240, which is unacceptable.

Fred

PS As Warren said, please stick to plaintext to make this
easier for everyone.

---

From: Hesham Soliman <[email-masked]>
Date: Thu, 17 May 2012 11:17:24 +1000

Can I ask a naive question on this, I haven't read the whole thread so
someone might have dismissed this already, but why can't the CPE advertise
a lower MTU if it knows that it's going to add tunnels ..etc? This way the
user's device will make sure whatever it sends (including et headers) fits
into the advertised MTU and the CPE can add the needed tunnels.

Just curious if this was already discussed.

Hesham

---

From: "Hemant Singh (shemant)" <[email-masked]>
Date: Wed, 16 May 2012 20:35:38 -0500

That is precisely what I and some of us have asked of in the thread.
The tunneled Internet connection from the CPE WAN to the SP is a
well-known connection to the CPE.  For example, the CPE knows the
connection is using the PPPOE protocol over 6rd.  PPPOE has a protocol
MTU of 1492 while the host behind the CPE has a default MTU of 1500.  So
the CPE computes the effective MTU and edits the TCP MSS option with the
effective MTU and thus a TCP client on the host also changes to the new
MTU.  The TCP MSS edit is also a stateless operation in the CPE.  

However, folks are asking, does the CPE do TCP packet inspection for all
time to come since the proposed text says the "CPE performs TCP MSS edit
on traversing the CPE".  My answer to such a question is, as far as the
tunneled Internet connections is up, the TCP packet inspection and TCP
MSS edit is enabled for the SYN packets.

Cheers,

Hemant

-----Original Message-----
From: [email-masked] [mailto:[email-masked]] On Behalf
Of Hesham Soliman
Sent: Wednesday, May 16, 2012 9:17 PM
To: IPv6 Operations
Subject: Re: [v6ops] proposed TCP MSS text for rfc6204bis

Can I ask a naive question on this, I haven't read the whole thread so
someone might have dismissed this already, but why can't the CPE
advertise
a lower MTU if it knows that it's going to add tunnels ..etc? This way
the
user's device will make sure whatever it sends (including et headers)
fits
into the advertised MTU and the CPE can add the needed tunnels.

Hesham

---

From: Cameron Byrne <[email-masked]>
Date: Wed, 16 May 2012 18:42:29 -0700

I largely agree that icmpv6 must be a solution. RA MTU perhaps too.

But my largest concern is ds-lite moving ipv4 in a tunnel across Docsis
with max mtu of 1522

While not yet in scope, MAP-E will have the same issues, no?

And mss adjusting  is the only tenable solution for this case  of ipv4 in
v6?

Sorry if i missed something obvious

Cb

---

From: Hesham Soliman <[email-masked]>
Date: Thu, 17 May 2012 13:41:21 +1000

-----Original Message-----
From: "Hemant Singh (shemant)" <[email-masked]>
Date: Wed, 16 May 2012 20:35:38 -0500
To: Hesham Soliman <[email-masked]>, IPv6 Operations
<[email-masked]>

>MTU.  The TCP MSS edit is also a stateless operation in the CPE.

=> But why edit the MSS instead of simply advertising that link MTU to the
client?
That way there is no need to edit the MSS.

Hesham

---

From: Mark Andrews <[email-masked]>
Date: Thu, 17 May 2012 15:34:17 +1000

> That way there is no need to edit the MSS.

Why should internal traffic be constrained by external mtu requirements?
Don't you think jumbo frames should be allowed to work?

---

From: Hesham Soliman <[email-masked]>
Date: Thu, 17 May 2012 15:42:32 +1000

>Don't you think jumbo frames should be allowed to work?

=> No I don't think that should be a factor in our decisions. We're
talking about deployments that are typically done in small home networks.
And we're talking about a small reduction in MTU (tunnel overhead). So I
don't think it's worth going for a hack like MSS rewrite to allow for
jumbo frames in this particular scenario.

We have a well defined tool for letting hosts know what the link MTU is,
and I don't think we should be resorting to muddle box hacks unless we
have to. In this home/small office scenario, it's perfectly fine to the
link MTU. In a more complex network with multiple routers within a site,
the leaf routers could advertise a higher MTU.

Hesham

---

From: Tassos Chatzithomaoglou <[email-masked]>
Date: Thu, 17 May 2012 09:53:37 +0300

I had expressed similar concerns in the past too.
http://www.ietf.org/mail-archive/web/v6ops/current/msg10716.html

Imho, i don't like the idea of CPE doing the MSS clamping.

--
Tassos

Hesham Soliman wrote on 17/5/2012 08:42:
>> In message<[email-masked]>, Hesham Soliman
>> writes:

---

From: "Hemant Singh (shemant)" <[email-masked]>
Date: Thu, 17 May 2012 04:25:14 -0500

Since there was no clear consensus on adding TCP MSS rewrite to
rfc6204bis, we released the -09 version yesterday with no TCP MSS
Rewrite.   

Joel, if the -09 goes to the IESG, please add Tim Winters to the
Acknowledgements list.  I realized I had to add his name but by then I
had already hit the "Post" button to ship the -09.

Thanks,

Hemant

-----Original Message-----
From: [email-masked] [mailto:[email-masked]] On Behalf
Of Tassos Chatzithomaoglou
Sent: Thursday, May 17, 2012 2:54 AM
To: Hesham Soliman
Cc: IPv6 Operations
Subject: Re: [v6ops] proposed TCP MSS text for rfc6204bis

--
Tassos

Hesham Soliman wrote on 17/5/2012 08:42:
>>>> MTU of 1492 while the host behind the CPE has a default MTU of
1500.
>>> So
>>>> the CPE computes the effective MTU and edits the TCP MSS option
with
>>> =>  But why edit the MSS instead of simply advertising that link MTU
to

---

From: Daniel Roesen <[email-masked]>
Date: Thu, 17 May 2012 13:33:22 +0200

> That way there is no need to edit the MSS.

Because you mess with LAN-internal communications then. I don't want a
router to mess with my home network's MTU around just because the WAN
link properties change.

My LAN has MTU 1500. If the WAN connection behind some random router has
a lower MTU, that's a standard scenario PMTUD deals with. And as we
operators know, PMTUD in the IPv4 Internet is FUBAR (a fact the whole
DSL industry has accepted and thus TCP MSS clamping is the norm), the
operator community WILL require vendors to do what it takes to make
DS-Lite as pain free to the enduser as possible: perform TCP MSS
clamping. It works for the whole PPPoE DSL industry "reasonably fine"
since over a decade.

BTW, also think about LANs with multiple routers present. Do you want to
have all of them mess around the LAN MTU in their RAs to their own
liking? One having a DS-Lite uplink (advertising LAN MTU 1460) and
another one having a 1500 octet uplink (advertising LAN MTU 1500)? And
your LAN-internal TCP sessions resetting all the time because hosts
adjust their interface MTU setting (and usually flap the link while
doing so) upon RA reception?

Don't. Mess. With. LAN MTU. It's horribly misguided to do so.

Best regards,
Daniel

---

From: Gert Doering <[email-masked]>
Date: Thu, 17 May 2012 13:42:52 +0200

Hi,

> We have a well defined tool for letting hosts know what the link MTU is,

Yes.  The *link* MTU.  Which has nothing to do whatsoever with "the minimum
MTU a packet might encounter somewhere on the Internet".

Gert Doering
        -- NetMaster

---

From: Hesham Soliman <[email-masked]>
Date: Thu, 17 May 2012 21:59:26 +1000

>router to mess with my home network's MTU around just because the WAN

=> But you're ok with it messing with most of the communication outside
the LAN?

>clamping. It works for the whole PPPoE DSL industry "reasonably fine"

=> How many home users do you know who would feel the pain of reducing
their internal LAN MTU by 10 bytes? I just don't find this reason serious
for the scenarios being discussed.
Engineering is about tradeoffs, the tradeoff here is to have a hack in the
GW for TCP only, Vs reducing the internal MTU by an insignificant amount
that none of the average users would ever care about or even care to know
what MTU is. 

>liking? 

=> As I mentioned in an earlier email, those were not the scenarios we're
discussing, if someone is setting such a network then it's a very
different scenario from the home network or small office networks we're
discussing which are typically single router networks.
If someone is setting up a multi-router network they can afford to be more
involved in the router config.

>One having a DS-Lite uplink (advertising LAN MTU 1460) and

=> I think it's misguided to have routers do this hack to avoid losing a
handful of bytes on the LAN's MTU.

Hesham

---

From: Hesham Soliman <[email-masked]>
Date: Thu, 17 May 2012 22:00:27 +1000

-----Original Message-----
From: Gert Doering <[email-masked]>
Date: Thu, 17 May 2012 13:42:52 +0200
To: Hesham Soliman <[email-masked]>
Cc: Mark Andrews <[email-masked]>, IPv6 Operations <[email-masked]>
Subject: Re: [v6ops] proposed TCP MSS text for rfc6204bis

>MTU a packet might encounter somewhere on the Internet".

=> Yes of course, so what's the comment? MSS editing has nothing to do
with any other MTU on the Internet either.

Hesham

---

From: [email-masked]
Date: Thu, 17 May 2012 09:04:03 -0400

>handful of bytes on the LAN's MTU.

100% agreement with Hesham here. 

While it's a noble goal to push for jumbo frames in the home, the
reality is that over 99.9% of Internet users could not care less. 
What they care about is having their "Internet connection" working. 
And some people on this list actually get paid at $dayjob to make 
sure this happens. 

<personnal experience>
I purchased a Cisco E4200v1 a while ago. Put it at home, turned on 
6RD. 6 minutes later, the tablet complained it can't download 
from Google Market. Fired a snoop: PMTUD problem, and the router doesn't 
send a lowered MTU in RAs either (heard the E4200v2 does it though). 
One minute later I disabled 6RD, although I am the one who actually 
deployed it for my ISP and pushing for customers to use it. I now 
use a router that sends a lowered MTU in RAs when 6RD is on and 
I lived happily everafter. 
</tranche de vie>

Bottom line: if it doesn't work right away out of the box for the 
average and not-so-average Internet user out there, we've collectively 
failed as a group to deliver specifications for an interoperable and
working Internet. Operational issues do matter. Just make it work. 

/JF 
</ranting>

---

From: "Templin, Fred L" <[email-masked]>
Date: Thu, 17 May 2012 08:39:18 -0700

Folks, it's not just about MSS clamping at the 6rd CPE;
the MTU of the whole IPv6 path has to be considered.
Let's say the CPE rewrites MSS to 1440, but there is a
1400 IPv6 link somewhere in the Internet path to the
final destination. The result is IPv6 PMTUD invocation,
but folks here seem to have doubts about the viability
of IPv6 PMTDU.

But let's look at that: if IPv6 PMTUD is broken then
tunnels or no tunnels we are sunk - if we do nothing
else,  then the only choice is to set all IPv6 host
interfaces to 1280. If we push for host deployment of
RFC4821, however, hosts get to live in the MTU diversity
world. Jumbogram support becomes a natural benefit of
that, but we can leave jumbograms out of the discussion
since the problem exists for *any* MTU greater than 1280.

Again, tunnels or no tunnels if we can't rely on IPv6
PMTUD the only recourse is for IPv6 hosts to set their
IPv6 interface MTUs to 1280 - unless they use RFC4821.

Turning back to 6rd tunnels, if the tunnel picks an
MTU and/or MSS clamping value that it *thinks* will
be safe, then there will always be cases when it
guesses wrong. If the 6rd egress tunnel endpoint has
a way of returning reliable IPv6 PMTUD feedback to the
6rd ingress tunnel endpoint, however, then the tunnel
can safely forget about MSS clamping and can even set
a larger MTU without fear of black holing.

We have an opportunity here to fix IPv6 PMTUD once
and for all rather than apply a best-guess band-aid
solution and pass the buck to someone else.

Fred

---

From: Daniel Roesen <[email-masked]>
Date: Thu, 17 May 2012 18:25:26 +0200

> solution and pass the buck to someone else.

For IPv6, perhaps. But not for IPv4, and that's what needs to be
considered for the DS-Lite case.

Best regards,
Daniel

---

From: "Templin, Fred L" <[email-masked]>
Date: Thu, 17 May 2012 09:28:41 -0700

Hi Daniel,

> Daniel Roesen
> Sent: Thursday, May 17, 2012 9:25 AM
> To: [email-masked]

AFAICT, RFC4821 works for IPv4 paths the same as it does
for IPv6 paths. Or, maybe you are concerned as to what
role the DS-Lite tunnel has to play in MTU handling. Can
you educate me on the issues?

Thanks - Fred

---

From: [email-masked]
Date: Thu, 17 May 2012 12:46:23 -0400

> guesses wrong.

Agreed. Whatever value is picked will always end up 
wrong in some cases. But speaking of 6RD, we can 
guess one that will be right most of the time (1480!). 

> If the 6rd egress tunnel endpoint has

Not sure I get your point. The issue with PMTUD is 
not within the tunnel domain most of the time. It is 
between the egress point and any host on the Internet
that blocks ICMPv6 (in short, any content provider 
behind a firewall blocking incoming ICMPv6). 

> solution and pass the buck to someone else.

There's no opportunity to fix IPv6 PMTUD here. Fixing 
it will be a constant and long-term battle to educate
content holders and other actors they have to let ICMPv6 
through or rate-limit it in a reasonable way (my opinion). 

/JF

---

From: "Templin, Fred L" <[email-masked]>
Date: Thu, 17 May 2012 10:06:50 -0700

From: [email-masked]
[mailto:[email-masked]] 
Sent: Thursday, May 17, 2012 9:46 AM
To: Templin, Fred L
Cc: [email-masked]; [email-masked]
Subject: Re: [v6ops] proposed TCP MSS text for rfc6204bis

> guess one that will be right most of the time (1480!).

6rd can only guess for a single IPv6 hop. There may be
many IPv6 hops in the path, but 6rd can only control
one of them. The only guiding standard we have is that
the IPv6 minMTU is 1280, so there is no way for 6rd to
assume anything more.

> behind a firewall blocking incoming ICMPv6).

The tunnel ingress needs to know when it needs to send
an ICMP PTB message back to a host within the 6rd site.
Choices are for the ingress to set a conservative MTU
(e.g., 1280) or set an unbounded MTU if it has a way
of getting reliable PMTUD feedback from the tunnel
egress. That way, hosts can expect to see 1500 or more
in most cases.  

> There's no opportunity to fix IPv6 PMTUD here. Fixing

Why not? We have outlined tools and strategies that
will get the job done.
 
> through or rate-limit it in a reasonable way (my opinion).

That kind of strategy is doomed to failure. We can't
tell the operators how to configure their links and
middleboxes - the right approach is to control the
hosts and to control the links that we have control
over (e.g., the 6rd tunnel).

Fred

PS Please can we stop with the html/rtf text and stick
   to plaintext?

---

From: [email-masked]
Date: Thu, 17 May 2012 13:23:18 -0400

> an ICMP PTB message back to a host within the 6rd site.

This is not an issue. In a residential scenario, the 
tunnel ingress is the CPE and it's usually a single 
L2 domain. There's nothing block PTB in that case, 
as long as the CPE sends it. 

You have to look at it the other way. When a content
provider starts sending large packets, the PTB will 
be generated at the 6RD BR back to the server. This 
is where it gets blocked IMO. 

> over (e.g., the 6rd tunnel).

Glad that you see it that way. However the issue is
not operators, this is a relatively small and educated 
crowd. The issue is the much larger crowd of "content
providers" out there, which is basically any native 
IPv6 host with a firewall. 

/JF

---

From: "Templin, Fred L" <[email-masked]>
Date: Thu, 17 May 2012 10:36:29 -0700

Looks like you ignored my request for plaintext
(thus making two-way conversation difficult) but
see below:

Sent: Thursday, May 17, 2012 10:23 AM

> as long as the CPE sends it.

Yes, the going-in assumption is that the ICMP PTB
coming from the CPE will not be blocked from reaching
the host within the CPE's site. But, the hosts would
be much better off if they saw a solid 1500 or larger
MTU. The only way the CPE's 6rd tunnel can do that is
by statelessly measuring the path MTU from the tunnel
ingress to tunnel egress. 

> is where it gets blocked IMO.

Sure. In that case, the CPE's 6rd tunnel endpoint
becomes the egress. Since the 6rd tunnel has absolutely
no control over the path MTU outside of the 6rd ISP,
however, the best the 6rd tunnel can do is 1280 - that
is, if we assume IPv6 PMTUD is broken. In that case,
the Internet servers should also implement RFC4821. 

> IPv6 host with a firewall.

The content providers deploy servers that will help
them make money. If they want to avoid complaints from
customers and loss of revenues, they will implement
RFC4821.

Thanks - Fred

---

From: [email-masked]
Date: Thu, 17 May 2012 14:18:48 -0400

"Templin, Fred L" <[email-masked]> a écrit sur 17/05/2012 
> (thus making two-way conversation difficult) but

Apologies. Working with a weird email client (is this better?). 

> The only way the CPE's 6rd tunnel can do that is

Statelessness and measuring are self-conflicting. No need
to reinvent PMTUD here, it usually works fine locally to 
the ISP toward the client. 

> customers and loss of revenues, they will implement

Service providers feel the pain first here. Residential
customers will call tech support before complaining to 
the content provider. 

We converged to the same conclusion here, content providers
have to make sure PTMUD works. We only disagree on the 
timing where they *most* will be compliant, which is 
"not any time soon" in my opinion. This is why the CPE
running 6RD should just lower the MTU in the RA and not 
rely on the content providers. 

/JF

---

From: "Templin, Fred L" <[email-masked]>
Date: Thu, 17 May 2012 12:41:49 -0700

Hi JF,

> From: [email-masked] [mailto:Jean-
> [email-masked]]
> Sent: Thursday, May 17, 2012 11:19 AM
> To: Templin, Fred L

Much better - thanks.

> Statelessness and measuring are self-conflicting. No need

No - not at all. If the egress sees a fragment of an
IPv6-in-IPv4 encapsulated packet, it can tell the
ingress how big the fragment was and the ingress
can use that size to statelessly report back to the
original source. Or, if the size of the fragment is
not helpful then the ingress consults a plateau
table and again statelessly reports back to the
original source.

> the ISP toward the client.

No reinvention; just a link-local adaptation to determine
the MTU of the link. It observes RFC1981 PMTUD in all
senses.

> rely on the content providers.

In an IPv6 PMTUD-broken world, lowering MTU is imperfect
unless you lower it all the way down to 1280. You'd be
better off leaving it at 1500 if you can get the 6rd
tunnel to do 1500.

Thanks - Fred

---

From: Gert Doering <[email-masked]>
Date: Thu, 17 May 2012 21:51:10 +0200

Hi,

> with any other MTU on the Internet either.

I'm not advocating MSS editing for IPv6.

IPv6 PMTU works - and where it breaks, it needs to be fixed.

For IPv4 (ds-lite etc.), MSS rewriting is unavoidable as IPv4 PMTU doesn't
work.

Gert Doering
        -- NetMaster

---

From: "Templin, Fred L" <[email-masked]>
Date: Thu, 17 May 2012 12:52:44 -0700

BTW, the reason for my delay in responding is that I am on
a confcall with a major Internet firewall product manufacturer.
I just asked him "does your product by default block ICMPv6
messages coming into and out of the site", and his answer
was "no - you cannot operate an IPv6 network w/o ICMPs".

Fred

> Sent: Thursday, May 17, 2012 12:42 PM
> To: [email-masked]

---

From: Mark Andrews <[email-masked]>
Date: Fri, 18 May 2012 10:26:37 +1000

> considered for the DS-Lite case.

IPv4 will still be in use for a decade or more.  The same fix works
for TCP for both IPv4 and IPv6.  Get the OS vendors to deploy and
back port RFC4821 support to all boxes that support IPv6.  This
will pickup mose IPv4 stacks in use as well.

Remember PMTUD works unless people deliberately stuff it up by
blocking ICMP messages which are part of the base protocol.  If
you start filtering you really need to know what you are doing.

A directed broadcast ping bomb from 2 decades ago is no excuse to
block ICMP today.  Firewall have selective ICMP filtering.

---

From: Shishio Tsuchiya <[email-masked]>
Date: Fri, 18 May 2012 10:14:51 +0900

> IPv6 PMTU works - and where it breaks, it needs to be fixed.

I hope so ,but there is problem analysis of IPv6 PMTD.

Path MTU Discovery‐ failure cases ‐
http://www.attn.jp/maz/p/t/tmp/pmtud-failure-cases-maz.pdf
RIPE60 Measurements of IPv6 Path MTU Discovery Behaviour
http://ripe60.ripe.net/presentations/Stasiewicz-Measurements_of_IPv6_Path_MTU_Discovery_Behaviour.pdf

World IPv6 day report,5%(11/241) websites may have IPv6 Path MTU Discovery problem.
http://hide.dnsalias.net/aaaa/worldipv6day.cgi

I think the data should not ignore,so CPE requirement document should mention MSS editing as *optional*.

Regard,
-Shishio

---

From: Lorenzo Colitti <[email-masked]>
Date: Fri, 18 May 2012 11:37:16 +0900

> you start filtering you really need to know what you are doing.

Lowering the MTU is not a reliability fix, it's a performance optimization.

Path MTU discovery needs to work. But path MTU discovery is slow - 1 RTT
for every new host you connect to.

In the case of 6rd, you know that 100% of hosts outside the home will incur
a packet too big, MTU=1480 response, every time, forever. So you
can improve performance by lowering the link MTU to 1480.

That will have an overhead of 1.3% on LAN-LAN communications compared to
1500 bytes, and will avoid the above latency impact.

Path MTU discovery is still needed for the case where there are MTU=1280
(or MTU=x, x<1480) links along the path.

What's wrong with this?

---

From: Hesham Soliman <[email-masked]>
Date: Fri, 18 May 2012 12:58:01 +1000

Nothing is wrong with this, it's a simple and logical way of doing things
IMO.

I find part of the problem here is that people seem to be confused about
the role of PMTUD.

In the context of the "MSS editing" thread, PMTUD _is_not_relevant_ to the
discussion. The discussion is about MSS editing by the CPE to avoid the
tunnelling impacts, which are known upfront and have nothing to do with
the impacts for the entire path.

I don't understand why people keep injecting PMTUD into the discussion
when we're talking about this specific case.

Hesham

-----Original Message-----
From: Lorenzo Colitti <[email-masked]>
Date: Fri, 18 May 2012 11:37:16 +0900
To: <[email-masked]>
Subject: Re: [v6ops] proposed TCP MSS text for rfc6204bis

>In the case of 6rd, you know that 100% of hosts outside the home will
>incur a packet too big, MTU=1480 response, every time, forever. So you

---

From: Lorenzo Colitti <[email-masked]>
Date: Fri, 18 May 2012 12:09:09 +0900

On Fri, May 18, 2012 at 11:58 AM, Hesham Soliman
<[email-masked]>wrote:

> In the context of the "MSS editing" thread, PMTUD _is_not_relevant_ to
> the discussion. The discussion is about MSS editing by the CPE to avoid
> the tunnelling impacts, which are known upfront and have nothing to do
> with the impacts for the entire path.
>

I think what people are saying is "don't rewrite MSS for IPv6 packets,
ever", and they end up talking about PMTUD because MSS rewriting is just a
hack to avoid PMTUD (and its possible failures).

---

From: Hesham Soliman <[email-masked]>
Date: Fri, 18 May 2012 13:40:09 +1000

><[email-masked]> wrote:
>
>
>ever", and they end up talking about PMTUD because MSS rewriting is just
>a hack to avoid PMTUD (and its possible failures).
=> It seems to be wrongly perceived as a "hack to avoid PMTUD". MSS
rewriting only considers a single link, the one connected to the CPE. So
it can't be seen as a hack to avoid PMTUD. Just a hack period.

Hesham

---

From: Mark Andrews <[email-masked]>
Date: Fri, 18 May 2012 13:50:56 +1000

In message <[email-masked]>
> hack to avoid PMTUD (and its possible failures).
 
Also if we want to record lowest path mtu we can do it with a per
hop IP{v6} option.  Doing this would help both TCP and UDP.  This
doesn't have to be sent for every packet.  TCP SYN/SYN ACK and the
first packet each way in a UDP exchange.

If the incoming and outgoing interfaces differ in MTU you set the
value to the lower of the lower of the two interfaces and the
existing value.  Routers when have equal sized MTU's can ignore the
option.

I'd much rather have support for something like this made manditory
for home routers than MSS re-write.

If we do add MSS re-write set a sunset date.  Kludges like this
should go away.

Mark

---

From: Brian E Carpenter <[email-masked]>
Date: Fri, 18 May 2012 08:38:51 +0100

> IPv6 PMTU works - and where it breaks, it needs to be fixed.

Yes, it works except where it doesn't.

Now is surely the time to develop a BCP on what everybody must do to make it work.

   Brian

---

From: "Templin, Fred L" <[email-masked]>
Date: Fri, 18 May 2012 08:03:40 -0700

> Path MTU discovery needs to work. But path MTU discovery
> is slow - 1 RTT for every new host you connect to.

s/1/1 or more/

RFC4821 *fixes* the RTT issue; RFC4821 TCPs will always
make progress w/o incurring a RTT even if there is a
restricting link in the path.

Fred

---

From: "Templin, Fred L" <[email-masked]>
Date: Fri, 18 May 2012 08:09:26 -0700

Hi Brian,

> Brian E Carpenter
> Sent: Friday, May 18, 2012 12:39 AM
> To: Gert Doering
> Cc: IPv6 Operations
> Subject: [v6ops] IPv6 PMTU works?? [proposed TCP MSS text for rfc6204bis]
> 

As I said before, if IPv6 PMTUD is broken then we're sunk;
set every link to 1280 and call it a day. But, we know that
that is an unacceptable outcome.
 
> Now is surely the time to develop a BCP on what everybody must do to make

There has been some progress related to this. Please see:

http://tools.ietf.org/html/draft-mlevy-ixp-jumboframes

which saw some momentum in the 'grow' wg back in the
November 2011 timeframe but now appears to be stalled.
Maybe interested parties should try to breathe some
new life into it.

Fred

---

From: "Templin, Fred L" <[email-masked]>
Date: Fri, 18 May 2012 08:29:27 -0700

> Shishio Tsuchiya
> Sent: Thursday, May 17, 2012 6:15 PM
> To: [email-masked]
> Cc: [email-masked]
> http://ripe60.ripe.net/presentations/Stasiewicz-
> Measurements_of_IPv6_Path_MTU_Discovery_Behaviour.pdf
> 

I was only able to read the second of these due to
firewall blockage. Can you send them to me in e-mail?

> mention MSS editing as *optional*.

That should not be necessary if the 6rd tunnel can
do 1500 or better.

Thanks - Fred

---

From: Nick Hilliard <[email-masked]>
Date: Fri, 18 May 2012 16:47:56 +0100

> Maybe interested parties should try to breathe some

In fact, Martin has been pushing this at various conferences outside the
IETF arena.  People seem to generally be positive about the draft.

However, speaking as an IXP operator who's attempted to get people
interested in jumbo frames, the problem with it is that the IXP
participants are very interested until the point where they have to sit
down in front of their routers and do something about it.  Then they stop
being interested.  Been there, got overwhelmed by the apathy.

Besides, the problem we're talking about here relates more to the MTU at
the edge rather than jumbo frames in the core.

Nick

---

From: "Templin, Fred L" <[email-masked]>
Date: Fri, 18 May 2012 08:53:10 -0700

Hi Nick,

> From: Nick Hilliard [mailto:[email-masked]]
> Sent: Friday, May 18, 2012 8:48 AM
> To: Templin, Fred L
> Cc: Brian E Carpenter; Gert Doering; IPv6 Operations
> Subject: Re: [v6ops] IPv6 PMTU works?? [proposed TCP MSS text for
> rfc6204bis]
> 

That's sad. Would it cut through the apathy if the document
were to say: "here's the *minimum* you must do in order to
avoid operational issues"?

> the edge rather than jumbo frames in the core.

I can easily imagine expanding the scope of the document
to cover the edge situations too.

Thanks - Fred

---

From: Nick Hilliard <[email-masked]>
Date: Fri, 18 May 2012 17:04:28 +0100

> avoid operational issues"?

the minimum includes a lot of monitoring of all point-to-point paths over
the network.  This is operationally difficult to do when you have
hash-based load balancing over multiple links, and you have no real way of
predetermining what path a particular packet will take.

Of course, when you have an MTU failure, it often causes silent
blackholing, and this tends only to be noticed by a small number of people
doing slightly fringe things, and the problem can only be debugged/fixed by
escalation to layer 3 engineering.

Why yes!  As you ask, I have been affected by this exact problem as a L3
transit customer in the last couple of months.  And it involved IPv6.

Nick

---

From: "Templin, Fred L" <[email-masked]>
Date: Fri, 18 May 2012 09:20:19 -0700

> Sent: Friday, May 18, 2012 9:04 AM
> To: Templin, Fred L

So, we either fix IPv6 PMTUD or we set all links to 1280;
is that what you're saying?

Fred

---

From: "Templin, Fred L" <[email-masked]>
Date: Fri, 18 May 2012 09:30:57 -0700

> That should not be necessary if the 6rd tunnel can

So, here's an idea (i.e., just an idea):

  1. set the MTU of all 6rd interfaces in the site to 1500
  2. for IPv6 packets exiting the 6rd CPE, the host accepts
     ICMPv6 PTB messages and reduces MSS if a PTB is received
  3. for IPv6 packets entering the 6rd BR from the Internet
     and destined to a host within a 6rd site, the BR does
     the following:

     a) drop the packet and send an ICMPv6 PTB if it is
        1501 or more
     b) encapsulate the packet in an IPv4 header and send
        it to the 6rd CPE if it is 1480 or less
     c) if the packet is between 1481 - 1500:
        - break it into 2 equal-sized pieces and insert a
          fragment header on both pieces
        - choose a random 32-bit value and write the value
          in the Identification field in both pieces
        - encapsulate both pieces in an IPv4 header and
          send them to the 6rd CPE

  4. the IPv6 host within the 6rd site gets to accept packets
     unfragmented packets in one piece and gets to reassemble
     fragmented packets that are no more than 1500

Fred

---

From: "Templin, Fred L" <[email-masked]>
Date: Fri, 18 May 2012 09:45:20 -0700

Let me make it simpler:

  1. set the MTU of all 6rd interfaces in the site to 1500
  2. for IPv6 packets to be admitted into the 6rd tunnel,
     do the following:

     a) drop the packet and send an ICMPv6 PTB if it is
        1501 or more
     b) encapsulate the packet in an IPv4 header and send
        it to the tunnel far end if it is 1480 or less
     c) if the packet is between 1481 - 1500:
        - break it into 2 equal-sized pieces and insert a
          fragment header on both pieces
        - choose a random 32-bit value and write the value
          in the Identification field in both pieces
        - encapsulate both pieces in an IPv4 header and
          send them to the tunnel far end
     d) the IPv6 destination host gets to reassemble if
        necessary

Fred

> Sent: Friday, May 18, 2012 9:31 AM
> To: Shishio Tsuchiya; [email-masked]
> Cc: [email-masked]

---

From: "Templin, Fred L" <[email-masked]>
Date: Fri, 18 May 2012 09:52:18 -0700

Sorry; one more iteration:

         it to the tunnel far end if it is 1280 or less
      c) if the packet is between 1281 - 1500:
 
Fred

> Sent: Friday, May 18, 2012 9:45 AM

---

From: Nick Hilliard <[email-masked]>
Date: Fri, 18 May 2012 18:01:29 +0100

> is that what you're saying?

basically yeah.

Nick

---

From: "Templin, Fred L" <[email-masked]>
Date: Fri, 18 May 2012 10:38:39 -0700

>    1. set the MTU of all 6rd interfaces in the site to 1500

Also, no need to set all 6rd interfaces at the same
time. This is incrementally deployable.

Fred

---

From: "Templin, Fred L" <[email-masked]>
Date: Fri, 18 May 2012 11:38:27 -0700

...and, here's the draft:

http://www.ietf.org/internet-drafts/draft-foo-v6ops-6rdmtu-00.txt

Fred

---

From: "Templin, Fred L" <[email-masked]>
Date: Fri, 18 May 2012 12:59:35 -0700

...and another:

http://www.ietf.org/internet-drafts/draft-bar-v6ops-ismtu-00.txt

Fred

---

From: Gert Doering <[email-masked]>
Date: Fri, 18 May 2012 22:29:23 +0200

Hi,

> http://www.ietf.org/internet-drafts/draft-foo-v6ops-6rdmtu-00.txt

Am I understanding this right that you're proposing to re-introduce
router fragmentation to IPv6?

I wasn't there when this was declared an absolute no-go some hundred
years ago, so I'm just staring in blank amazement...

Gert Doering
        -- Operator

---

From: "Templin, Fred L" <[email-masked]>
Date: Fri, 18 May 2012 13:32:40 -0700

> From: Gert Doering [mailto:[email-masked]]
> Sent: Friday, May 18, 2012 1:29 PM
> To: Templin, Fred L
> Cc: Shishio Tsuchiya; [email-masked]; [email-masked]

Well, for some special cases of router, yes.
 
> years ago, so I'm just staring in blank amazement...

It's heresy, I know. But, do you see anything wrong
with it?

Thanks - Fred

---

From: Randy Bush <[email-masked]>
Date: Sat, 19 May 2012 08:52:49 +0900

> In fact, Martin has been pushing this at various conferences outside
> the IETF arena.

[ not judging this rightly or wrongly ]

it is an ieee standard, and that is the only place where it can be
meaningfully pushed.  and they have pretty firmly said "hell no" for
over a decade.

so configure it with a friend in the privacy of your own link or within
your own datacenter.

randy

---

From: Hesham Soliman <[email-masked]>
Date: Sat, 19 May 2012 12:38:38 +1000

Fred, 

Just for the record, you're not introducing router fragmentation. The
router here is the originator of the new packet, so you're not breaking
anything in IPv6 in your idea, which I don't have an opinion on btw.

Hesham

---

From: Brian E Carpenter <[email-masked]>
Date: Sat, 19 May 2012 07:57:26 +0100

> It's heresy, I know. But, do you see anything wrong

A more interesting heresy would be to remove fragmentation from
IPv6 entirely. There's already a rule that any layer 2 used for
IPv6 must carry a payload of at least 1280 bytes:

   IPv6 requires that every link in the internet have an MTU of 1280
   octets or greater.  On any link that cannot convey a 1280-octet
   packet in one piece, link-specific fragmentation and reassembly must
   be provided at a layer below IPv6.
   [RFC2460 section 5]

That is necessary and sufficient to make the Internet work. Can somebody
remember why we added the complication of fragmentation?

    Brian

---

From: Gert Doering <[email-masked]>
Date: Sat, 19 May 2012 09:58:49 +0200

Hi,

> anything in IPv6 in your idea, which I don't have an opinion on btw.

The router is creating a new *IPv4* packet, and packing the existing 
IPv6 packet into it.  It's not creating a new IPv6 packet.

If the 6rd router were to be fragmenting the external *IPv4* packet,
there would not be anything spectacular about this proposal.

Gert Doering
        -- NetMaster

---

From: Gert Doering <[email-masked]>
Date: Sat, 19 May 2012 10:00:06 +0200

Hi,

> remember why we added the complication of fragmentation?

UDP packets larger than 1280 bytes, and link-layers that are a) limited in
packet size, and b) have no reasonable way to do on-link fragmentation and
reassembly, like "Ethernet".

Gert Doering
        -- NetMaster

---

From: Brian E Carpenter <[email-masked]>
Date: Sat, 19 May 2012 11:27:47 +0100

On 2012-05-19 09:00, Gert Doering wrote:
> UDP packets larger than 1280 bytes

Don't do that!

> and link-layers that are a) limited in

That is *specifically* excluded by the text in 2460: if you have such a
link, you must insert an adaptation layer - it is not covered by layer 3.

   Brian

---

From: Nick Hilliard <[email-masked]>
Date: Sat, 19 May 2012 11:41:09 +0100

> meaningfully pushed.  and they have pretty firmly said "hell no" for

they may have said no, however that hasn't stopped pretty much every nic
manufacturer in the world supporting 9k mtu on their devices, even on
trashy low-end kit.

> so configure it with a friend in the privacy of your own link or within

I think you're right here.  There are some technologies which don't tend to
work well on interconnection links due to different expectations on each
side.  This is one of them.

fwiw, inex still has a 9k mtu vlan (full of tumbleweed) but we're very
clear that our primary peering vlans are 1500 only.  We may try to
resuscitate the 9k mtu vlan at some stage in the future.

anyway, this is well off topic.

Nick

---

From: [email-masked]
Date: Sat, 19 May 2012 14:19:06 +0200 (CEST)

> > UDP packets larger than 1280 bytes

It sounds like you're saying that we should expect IPv6 to work less
well than IPv4. Surely I'm misunderstanding?

> link, you must insert an adaptation layer - it is not covered by layer 3.

So we shouldn't be able to send large IPv6 datagrams on Ethernet. Or?

I think these points are probably down in the noise when it comes to
actually getting IPv6 deployed. Nevertheless, they are important to
some people.

Steinar Haug, Nethelp consulting, [email-masked]

---

From: Brian E Carpenter <[email-masked]>
Date: Sat, 19 May 2012 15:01:48 +0100

> well than IPv4. Surely I'm misunderstanding?

What I am saying (and remember this is heresy) is that apps that assume
that large UDP packets will arrive, without any kind of trial-and-error
approach, are badly designed. There's always going to be a finite PMTU.

> So we shouldn't be able to send large IPv6 datagrams on Ethernet. Or?

If you want to send packets of arbitrary size, in any environment where
PMTUD is impossible or fails, won't you need to always include
a fragmentation header in every packet greater than 1280?

> actually getting IPv6 deployed. Nevertheless, they are important to

Well, I was an advocate for RFC 2675 since it seemed important where
I worked at that time:
"The Jumbo Payload option is relevant only for IPv6 nodes that may be
attached to links with a link MTU greater than 65,575 octets (that
is, 65,535 + 40, where 40 octets is the size of the IPv6 header)...
The Jumbo Payload option must not be used in a packet that carries a
Fragment header."

In other words people who need really big packets know that they must
avoid fragmentation.

Getting back to the topic at hand, I've reached the conclusion (by a
sort of reductio ad absurdum) that expecting an off-the-shelf CPE
to fix this problem is unreasonable.

   Brian

---

From: Gert Doering <[email-masked]>
Date: Sat, 19 May 2012 22:16:12 +0200

Hi,

> > UDP packets larger than 1280 bytes

Tell that to the DNS people.  They seem to really like not-using-TCP.

> link, you must insert an adaptation layer - it is not covered by layer 3.

Usually "Ethernet" is not considered a medium that needs an adaption
layer...

Gert Doering
        -- NetMaster

---

From: Randy Bush <[email-masked]>
Date: Sun, 20 May 2012 06:26:43 +0900

> If you want to send packets of arbitrary size, in any environment
> where PMTUD is impossible or fails, won't you need to always include

see discussion of jumbo frames, commonly 4k or 9k, between consenting
adults on known links

randy

---

From: Brian E Carpenter <[email-masked]>
Date: Sun, 20 May 2012 08:03:15 +0100

> see discussion of jumbo frames, commonly 4k or 9k, between consenting

Yes indeed, but that isn't the general case. Across the open Internet,
I think we have the situation I described.

> Tell that to the DNS people.  They seem to really like not-using-TCP.

Yes, but I understand that DNSSEC more or less dooms that plan anyway.

However, I thinks it's true that the only fail-safe solution is to
include a frag header if you need to send UDP >1280.

   Brian

    Brian

---

From: JINMEI Tatuya / 神明達哉 <[email-masked]>
Date: Sun, 20 May 2012 00:53:12 -0700

At Sun, 20 May 2012 08:03:15 +0100,

> Yes, but I understand that DNSSEC more or less dooms that plan anyway.

I may misunderstand the context, but modern and sensible DNS server
implementations generally enable the IPV6_USE_MIN_MTU socket option.

> include a frag header if you need to send UDP >1280.

---
JINMEI, Tatuya
Internet Systems Consortium, Inc.

---

From: Brian E Carpenter <[email-masked]>
Date: Sun, 20 May 2012 09:36:57 +0100

> implementations generally enable the IPV6_USE_MIN_MTU socket option.

OK...

>> include a frag header if you need to send UDP >1280.

As I understand it, that is triggered automatically by IPV6_USE_MIN_MTU.
So any UDP application that *doesn't* set IPV6_USE_MIN_MTU = 1 is
at risk of failure to some destinations, unless it performs its own
transport-layer PMTUD somehow.

However, we are well off the MSS topic so I will shut up now.

   Brian

---

From: Mark Andrews <[email-masked]>
Date: Mon, 21 May 2012 00:04:20 +1000

> include a frag header if you need to send UDP >1280.

For DNS we just fragment at 1280 using IPV6_USE_MIN_MTU.   We were
thinking about this back in 1998 (draft-ietf-ipngwg-bsd-frag-00.txt)
which was rolled into the advanced socket api.  It took a few more
years than I would have liked to become RFC and for implementations
to be available.  EDNS was already being developed back then and
it was obvious that PMTUD wouldn't work for large nameservers even
if they got the ICMPv6 PTBs.

For DNS there is little to be gained by trying to send any bigger
packets.

YMMV for other UDP based protocols.

Mark

---

From: Warren Kumari <[email-masked]>
Date: Mon, 21 May 2012 10:49:02 -0400

[ Top posting / meta questions ]

So, both my memory and my google-foo are failing me…

Can anyone remember *why* the v6 in MTU is 1280 (and please don't say "Because the RFC says so!"…)

I have some niggling voice in the back of my head making 3GPP noises, but…

W

-- 
With Feudalism, it's your Count that votes.

---

From: Simon Perreault <[email-masked]>
Date: Tue, 22 May 2012 10:26:22 -0400

> Can anyone remember *why* the v6 in MTU is 1280 (and please don't say "Because the RFC says so!"…)

See this thread:
http://www.ietf.org/mail-archive/web/v6ops/current/msg11892.html

Simon

---

From: Rémi Després <[email-masked]>
Date: Tue, 22 May 2012 17:42:55 +0200

Hi Warren,

> Can anyone remember *why* the v6 in MTU is 1280 (and please don't say "Because the RFC says so!"…)

In the excellent French book in which I learned IPv6 in 2003 (livre.g6.asso.fr/index.php/Format_du_paquet_IPv6), here is what I learned:
"Choice of 1280 as minimum MTU is to permit IPv6 tunneling. 1500 octets is the generally admitted link MTU, that imposed by Ethernet." 

As this permits multiple tunneling layers without fragmentation on the most typical links, and limits to a moderate ratio waste of bandwidth on 1500-octet remote links, this is IMHO a valuable choice.

RD

---

From: Warren Kumari <[email-masked]>
Date: Tue, 22 May 2012 13:16:23 -0400

> As this permits multiple tunneling layers without fragmentation on the most typical links, and limits to a moderate ratio waste of bandwidth on 1500-octet remote links, this is IMHO a valuable choi…

Yes,  I get that it is to accommodate tunnel overhead (and still fit in an Ethernet frame),  I was more wondering why the overhead was chosen as 220 bytes and not 216 or 247 or 196.324.

I received this off-list (including with permissions):

"Hi Warren,
    Off-list since I cannot find a definitive reference at this point.  The 1280 number was codified in RFC 2460 based on a back of the envelope calculation by Steve Deering either before or during the Fall 1997 IETF meeting.  Starting with the Ethernet MTU of 1500, he subtracted the size of : ethernet header, IPv6 header, IPv4 header (tunneling), and several possible extension headers.  That put the number at 1280 (with some rounding to a 64-bit boundary)."

W

---

From: "Rajiv Asati (rajiva)" <[email-masked]>
Date: Tue, 22 May 2012 22:47:09 -0500

Why would Ethernet header size be subtracted in that equation, since RFC894 (in 1984) had already specified 1500B size IP packet over ethernet? I am puzzled.

http://tools.ietf.org/html/rfc894
..
   The minimum length of the data field of a packet sent over an
   Ethernet is 1500 octets, thus the maximum length of an IP datagram
   sent over an Ethernet is 1500 octets.  Implementations are encouraged
..

Cheers,
Rajiv

> Sent: Tuesday, May 22, 2012 1:16 PM
> To: Rémi Després
> Cc: [email-masked]
> >> So, both my memory and my google-foo are failing me...
> >>
> >> Can anyone remember *why* the v6 in MTU is 1280 (and please don't say
> >> "Because the RFC says so!"...)
> >
> > In the excellent French book in which I learned IPv6 in 2003
> (livre.g6.asso.fr/index.php/Format_du_paquet_IPv6), here is what I
> learned:
> > "Choice of 1280 as minimum MTU is to permit IPv6 tunneling. 1500 octets is
> the generally admitted link MTU, that imposed by Ethernet."
> >
> > As this permits multiple tunneling layers without fragmentation on the
> most typical links, and limits to a moderate ratio waste of bandwidth on
> 1500-octet remote links, this is IMHO a valuable choice.
> 
> Yes,  I get that it is to accommodate tunnel overhead (and still fit in an
> Ethernet frame),  I was more wondering why the overhead was chosen as
> 220 bytes and not 216 or 247 or 196.324.
> 
>     Off-list since I cannot find a definitive reference at this point.  The 1280
> number was codified in RFC 2460 based on a back of the envelope
> calculation by Steve Deering either before or during the Fall 1997 IETF
> meeting.  Starting with the Ethernet MTU of 1500, he subtracted the size of :
> ethernet header, IPv6 header, IPv4 header (tunneling), and several possible
> extension headers.  That put the number at 1280 (with some rounding to a
> 64-bit boundary)."
> 
> 
> W
> 
> 
> >
> > RD
> >
> >
> >
> >> I have some niggling voice in the back of my head making 3GPP noises,
> >> but...
> >>
> >> W
> >>
> >>>>>> where PMTUD is impossible or fails, won't you need to always
> >>>>>> include a fragmentation header in every packet greater than 1280?
> >>>>>
> >>>>> see discussion of jumbo frames, commonly 4k or 9k, between
> >>>>> consenting adults on known links
> >>>>
> >>>> Yes indeed, but that isn't the general case. Across the open
> >>>> Internet, I think we have the situation I described.
> >>>>
> >>>>>>> UDP packets larger than 1280 bytes Don't do that!
> >>>>>
> >>> to be available.  EDNS was already being developed back then and it
> >>> was obvious that PMTUD wouldn't work for large nameservers even if
> >>> they got the ICMPv6 PTBs.
> >>>

---

From: "Templin, Fred L" <[email-masked]>
Date: Wed, 23 May 2012 07:38:59 -0700

> "Because the RFC says so!"...)

You have to reach way back into history to find the
reasoning. The first proposal of moving from 576 to
1280 was from Steve Deering in November 1997. Go to:

https://www.sixxs.net/archive/docs/ipng-archives/ipng.199711

and search for:

	(IPng 4802) increasing the IPv6 minimum MTU

Remember that IPv6 is a very old protocol, and the
history goes way back. Digging the archives can be
very revealing.

Fred

> Warren Kumari
> Sent: Monday, May 21, 2012 7:49 AM
> To: Mark Andrews
> Cc: [email-masked]
> I have some niggling voice in the back of my head making 3GPP noises, but...
> 
> W
>

---

From: Simon Perreault <[email-masked]>
Date: Wed, 23 May 2012 10:50:03 -0400

On 2012-05-23 10:38, Templin, Fred L wrote:
> Go to:

Direct link for the lazy:
http://web.archive.org/web/20001210152900/http://www.cs-ipv6.lancs.ac.uk/ipv6/mail-archive/IPng/1997-12/0052.html

Simon