Subject: Re: [v6ops] new draft: draft-byrne-v6ops-64share

From: Mark ZZZ Smith <[email-masked]>
Date: Mon, 1 Oct 2012 02:45:43 -0700 (PDT)

----- Original Message -----
> From: Daniel Roesen <[email-masked]>
> To: [email-masked]; "[email-masked]" <[email-masked]>
> Cc: 
> Sent: Monday, 1 October 2012 10:08 AM
> Subject: Re: [v6ops] new draft: draft-byrne-v6ops-64share
> 
> On Sat, Sep 29, 2012 at 02:50:49PM -0700, Cameron Byrne wrote:
>>  On Sat, Sep 29, 2012 at 2:27 PM, Eric Vyncke (evyncke)
>>  <[email-masked]> wrote:
>>  > Isn't it easier to copy the MTU option from the RA received by the 
> handset to the RA sent on the LAN? Of course, when there is no MTU option in the 
> received RA on the WAN link, then, a SHOULD set MTU option to 1480 seems good
>> 
>>  Yes, this will be the logic that we will bring to the text of
>>  draft-binet-v6ops-cellular-host-reqs-rfc3316update
> 
> I strongly disagree with that. Let PMTUD do its work, it's there
> for that. I really really don't want to see routers on a LAN
> arbitrarily mucking around with the LAN MTU to work around random
> problems of upstream infrastructure.
> 
> This artificial MTU lowering in the LAN also breaks connectivity with
> nodes statically configured, disregarding RAs. Suddenly you have nodes
> with interface MTU=1500 trying to communicated with other nodes on the
> same LAN with other, RA-derrived lower MTUs...
> 

RAs are for more than just announcing a default router, they're for all network layer parameters (perhaps "Network Layer Configuration Protocol" might have been a better name than Router Advertisements). Even a statically addressed host should be listening to them, as they'll inform it of all of the on-link prefixes, including the prefix it's address comes from. Unlike IPv4, in IPv6, a host does not assume that other addresses that fall within it's configured prefixes are on-link - they learn that information from the Prefix Information Option in the RAs. A host with a static IPv6 address that is ignoring RAs will send all of it's traffic to it's default router, regardless of whether the destination address falls within the prefix that the static address is assigned from. This could be useful where the router is also performs some sort of policy enforcement or traffic accounting.

It's a subtle but significant difference between IPv4 and IPv6. See RFC5942, "IPv6 Subnet Model: The Relationship between Links and Subnet Prefixes".

Regards,
Mark.

---

From: Daniel Roesen <[email-masked]>
Date: Mon, 1 Oct 2012 11:55:16 +0200

On Mon, Oct 01, 2012 at 03:17:33PM +0900, Lorenzo Colitti wrote:
> On Mon, Oct 1, 2012 at 9:08 AM, Daniel Roesen <[email-masked]> wrote:
> 
> Bad idea. PMTUD causes a ~1 RTT latency hit to every new server you talk to.

Well, that's life. If you want to avoid PMTUD at all cost, restrict
everything to 1280 in the first place.

BTW, ~1 RTT between which nodes? Certainly not between the user's client
and the server being connected, but just between the user's client and
the mobile device.

If you really care about delay, MSS clamping is an option which isn't
intrusive to the LAN as it affects only connections via the mobile
gateway.

> Why do you need 1500 anyway?

Pardon?

> So that devices talking to each other via the phone's hotspot (!)

Misconception. We're discussing of messing with RAs to the LAN. Wether
that's a WiFi hotspot, or a wired link, or a WiFi link to a WiFi bridge
into a wired LAN doesn't matter.

> can use 1500-byte packets instead of 1480-byte packets,
> thus gaining a whopping 0.05% performance boost? ((1440÷1500) / (1420÷1480)
> - 1)

It's not a matter of performance. It's a matter of mucking around with a
shared network's MTU dependent on the MTU of some uplink WAN interface.
That's just wrong. The mobile device isn't necessarily the "boss on the
link" but just a participant offering default routing (at time where it
has a working WAN link, that is).

> That's really not worth the latency hit.

I beg to differ, for reasons of technical correctness, not throughput.

What happens when some other router exists in the LAN, with a different
notion of LAN link MTU? What about endpoints that do ignore MTU received
in RAs? (I think I observed several that flat-out do ignore MTU)

> Statically configured addresses on the tethering hotspot? How would that
> even happen, given that in most cases your prefix is dynamically assigned?
> If you don't know the prefix in advance, you can't configure the address.

"most cases" - what about business connections with static prefixes? Oh,
and it's not a handset, but a SOHO backup router.

You have a specific deployment scenario in mind, but your approach
breaks others. Not good.

Perhaps I'm confused? Do I miss something?

Best regards,
Daniel

-- 
CLUE-RIPE -- Jabber: [email-masked] -- dr@IRCnet -- PGP: 0xA85C8AA0

---

From: Daniel Roesen <[email-masked]>
Date: Mon, 1 Oct 2012 12:04:18 +0200

On Sun, Sep 30, 2012 at 06:49:52PM -0700, Cameron Byrne wrote:
> That said, it is not "arbitrarily mucking around with the LAN MTU to
> work around random problems"
>
> It is in fact deterministicly working around a systemic problem in
> 3GPP network using the correct IETF tools to do so: RA MTU info.

MTUs were never meant to get "propagated" but are a link-specific
attribute. Tying link A MTU to link B MTU is just wrong.

> Keep in mind, this problem is "solved" today by "adjusting" the MSS in
> HTTP proxies, which are also common in 3GPP networks.  Nobody asked
> for the IETF's permission for this pervasive yet effective technique
> (which is also common in DSL network, and will soon popup in DS-lite
> networks... but IPv4 does not have RA to communicate the MTU)

I'm well aware of that, having deployed DS-Lite in residential access
network with tunnel payload MTU 1460 and LAN MTU 1500, using MSS
clamping to optimize for Lorenzo's considerations. Works beautifully.

MSS clamping is a far "better" hack for the problem discussed than
mucking with LAN MTUs.

> Yes, PMTU will discover the correct MTU.  But why not start out in the
> right place?

Because it's only the "right place" for WAN-facing traffic, not for
traffic within the LAN.

> If the 3GPP network operator knows that the MTU is of
> the WAN link is 1440, why not cascade the information down via RA ?

Again, because you're influencing intra-LAN communication as well.

> PMTU will be quicker and more effective in a LAN than a WAN if this
> problem occurs.

Can you elaborate? Two hosts trying to communicate with different MTU
settings on their interfaces will just lead to traffic blackholing, as
the node with the smaller MTU setting will just discard "too large"
packets. There is no router which will send "packet too big, max XYZ".

Best regards,
Daniel

---

From: Tore Anderson <[email-masked]>
Date: Mon, 01 Oct 2012 12:15:30 +0200

* Daniel Roesen

> packets. There is no router which will send "packet too big, max XYZ".

I think you are confusing MTU with MRU here. A host with a interface MTU
of, say, 1280 may very well be capable of receiving and processing 1500
byte large packets. MTU is the Maximum *Transmission* Unit, it has
nothing to do with receiving behaviour.

-- 
Tore Anderson
Redpill Linpro AS - http://www.redpill-linpro.com

---

From: Mikael Abrahamsson <[email-masked]>
Date: Mon, 1 Oct 2012 12:26:38 +0200 (CEST)

> and the server being connected, but just between the user's client and 

User device sends SYN with MTU 1500 based MSS. After a while server will 
most likely start sending 1500 byte frames, which will get PMTUD PTB 
message from the GGSN (the network facing device which does GTP 
encapsulation). That's where the problematic RTT is.

> intrusive to the LAN as it affects only connections via the mobile 

But it only helps TCP, not everything else.

> has a working WAN link, that is).

I wish we had a way to put a separate MTU on just the default route being 
installed due to seeing RA, instead of just being able to set MTU on the 
LAN.

> in RAs? (I think I observed several that flat-out do ignore MTU)

Then we will see GTP fragmentation, but at least most cases will mean we 
will not see GTP fragmentation. The aim is to keep it down, not eliminate 
it completely.

-- 
Mikael Abrahamsson    email: [email-masked]

---

From: Daniel Roesen <[email-masked]>
Date: Mon, 1 Oct 2012 13:55:24 +0200

> nothing to do with receiving behaviour.

Do you know any popular host implementation distinguishing MTU and MRU?
Last time I looked at the Linux Ethernet+IP stack, it didn't differentiate,
as far as I can remember. Anything above MTU setting gets dropped. In
fact, the Linux kernel doesn't even have the infrastructure in the
netdev struct to distinguish both.

Oh, and devices happen to reset their interface when MTU changes (I
confirmed in code of the e1000 driver), sometimes results in link flaps,
usually breaking all higher-level sessions crossing that link that way.

Anyway, you cannot rely on a node being able to receive packets/frames
that are larger than the configured MTU.

Best regards,
Daniel

---

From: Daniel Roesen <[email-masked]>
Date: Mon, 1 Oct 2012 14:01:57 +0200

On Mon, Oct 01, 2012 at 12:26:38PM +0200, Mikael Abrahamsson wrote:
>> and the server being connected, but just between the user's client and the 
>> mobile device.
>

Oh, you're completely right. OK, so it's server-GGSN RTT, correct.

> But it only helps TCP, not everything else.

LAN MTU trickery doesn't either.

> installed due to seeing RA, instead of just being able to set MTU on the 

Indeed. But RA MTU signalling is about a _link_, not a path (like the
default route). Don't abuse it for that, pretty please. :)

> will not see GTP fragmentation. The aim is to keep it down, not eliminate 

MSS clamping will do that just fine. Or do you see relevant amount of
non-TCP traffic with relevant high packet sizes in those mobile
networks?

Best regards,
Daniel

---

From: Mikael Abrahamsson <[email-masked]>
Date: Mon, 1 Oct 2012 14:26:57 +0200 (CEST)

> default route). Don't abuse it for that, pretty please. :)

I wouldn't call it abuse, I think it's the least bad option. If RA had a 
default route MTU option, we wouldn't need to lower LAN MTU.

> non-TCP traffic with relevant high packet sizes in those mobile 

Isn't MSS clamping even worse hackery? Why do you prefer MSS clamping 
above changing the MTU? If we change the MTU, then all the standard tools 
(PMTUD) should work as designed. If it doesn't, then why not fix that 
instead?

---

From: Tore Anderson <[email-masked]>
Date: Mon, 01 Oct 2012 14:44:18 +0200

* Daniel Roesen

> netdev struct to distinguish both.

192.168.0.42 is running Linux 3.5.4, Fedora 17. "ping -M do -s 1472
192.178.0.42" running on the 192.158.0.2 host - starting out with an
MTU of 1500 on both hosts:

14:26:35.075642 IP (tos 0x0, ttl 64, id 0, offset 0, flags [DF], proto ICMP (1), length 1500)
    192.168.0.2 > 192.168.0.42: ICMP echo request, id 10293, seq 114, length 1480
14:26:35.075677 IP (tos 0x0, ttl 64, id 26631, offset 0, flags [none], proto ICMP (1), length 1500)
    192.168.0.42 > 192.168.0.2: ICMP echo reply, id 10293, seq 114, length 1480

[running "ip link set mtu 1280 dev em1" on 192.168.0.42 here]

14:26:41.077492 IP (tos 0x0, ttl 64, id 0, offset 0, flags [DF], proto ICMP (1), length 1500)
    192.168.0.2 > 192.168.0.42: ICMP echo request, id 10293, seq 120, length 1480
14:26:41.077538 IP (tos 0x0, ttl 64, id 26518, offset 0, flags [+], proto ICMP (1), length 1276)
    192.168.0.42 > 192.168.0.2: ICMP echo reply, id 10293, seq 120, length 1256
14:26:41.077617 IP (tos 0x0, ttl 64, id 26518, offset 1256, flags [none], proto ICMP (1), length 244)
    192.168.0.42 > 192.168.0.2: icmp
[...]
14:26:46.076060 IP (tos 0x0, ttl 64, id 0, offset 0, flags [DF], proto ICMP (1), length 1500)
    192.168.0.2 > 192.168.0.42: ICMP echo request, id 10293, seq 125, length 1480
14:26:46.076104 IP (tos 0x0, ttl 64, id 26523, offset 0, flags [+], proto ICMP (1), length 1276)
    192.168.0.42 > 192.168.0.2: ICMP echo reply, id 10293, seq 125, length 1256
14:26:46.076110 IP (tos 0x0, ttl 64, id 26523, offset 1256, flags [none], proto ICMP (1), length 244)
    192.168.0.42 > 192.168.0.2: icmp

[running "ip link set mtu 1500 dev em1" on 192.168.0.42 here]

14:26:51.075830 IP (tos 0x0, ttl 64, id 0, offset 0, flags [DF], proto ICMP (1), length 1500)
    192.168.0.2 > 192.168.0.42: ICMP echo request, id 10293, seq 130, length 1480
14:26:51.075883 IP (tos 0x0, ttl 64, id 26518, offset 0, flags [none], proto ICMP (1), length 1500)
    192.168.0.42 > 192.168.0.2: ICMP echo reply, id 10293, seq 130, length 1480

So, as you can see, reducing the MTU did not in any way impair .42's
ability to receive, process, and respond to .2's full-sized packets.
However, it had to start fragmenting its responses, which is as
expected.

> usually breaking all higher-level sessions crossing that link that way.

Four packets were lost in my tcpdump, but this wasn't something the higher
level protocols didn't deal with like any other packet loss. My Spotify
stream didn't skip a beat, for example.

.42 has a Intel 82579LM GE LOM NIC btw (desktop/consumer grade stuff).

> that are larger than the configured MTU.

Perhaps not. But you can certainly not rely on the configured MTU having
anything to do with the effective MRU value, as demonstrated above. I'm
guessing that in the common case, the MRU is determined by limitations in
the chipset. The MTU, on the other hand, is a setting in the OS networking
stack (which obviously may not be set arbitrarily high without eventually
running into some chipset limitation).

---

From: Rémi Després <[email-masked]>
Date: Mon, 1 Oct 2012 15:34:29 +0200

Hi, Daniel, Mikael, 

2012-10-01 12:26, Mikael Abrahamsson :

> ...
>> If you really care about delay, MSS clamping is an option which isn't intrusive to the LAN as it affects only connections via the mobile gateway.
> 

Since it does help TCP and doesn't hurt anything, it's IMHO worth recommending.

FYI, something was proposed about this in May, ref. www.ietf.org/mail-archive/web/v6ops/current/msg13015.html, namely:
"In any node, before entering of just after leaving a link whose MTU is the same in both directions (Ethernet, 6rd, DS-lite, ...), if the MSS advertised in a received TCP SYN packet exceeds the link MTU minus the TCP/IP header length (40 octets in IPv4, 60 in IPv6), reduce it to this value, and adjust IP and TCP checksums accordingly."

In principle, I can't any longer afford to be active in v6ops, but still look at some of the exchanged mails and may provide brief comments. Dan Wing is added as cc because he privately said in May he liked this proposed behavior.

Regards,
RD

---

From: "Templin, Fred L" <[email-masked]>
Date: Mon, 1 Oct 2012 08:59:22 -0700

> -----Original Message-----
> From: [email-masked] [mailto:[email-masked]] On Behalf Of
> Rémi Després
> Sent: Monday, October 01, 2012 6:34 AM
> To: Daniel Roesen; Mikael Abrahamsson
> Cc: v6ops WG
> Subject: Re: [v6ops] LAN vs WAN MTUs (was new draft: draft-byrne-v6ops-
> 64share)
> 
> Hi, Daniel, Mikael,
> 
> FYI, something was proposed about this in May, ref. www.ietf.org/mail-
> archive/web/v6ops/current/msg13015.html, namely:
> "In any node, before entering of just after leaving a link whose MTU is
> the same in both directions (Ethernet, 6rd, DS-lite, ...), if the MSS
> advertised in a received TCP SYN packet exceeds the link MTU minus the
> TCP/IP header length (40 octets in IPv4, 60 in IPv6), reduce it to this
> value, and adjust IP and TCP checksums accordingly."
> 
> In principle, I can't any longer afford to be active in v6ops, but still
> look at some of the exchanged mails and may provide brief comments. Dan
> Wing is added as cc because he privately said in May he liked this
> proposed behavior.

The alternative of course is to fix tunnel MTU for good, as
specified here:

https://datatracker.ietf.org/doc/draft-generic-v6ops-tunmtu/
https://datatracker.ietf.org/doc/draft-generic-6man-tunfrag/
https://datatracker.ietf.org/doc/draft-templin-intarea-seal/

Thanks - Fred
[email-masked]

> Regards,
> RD
> 
> _______________________________________________
> v6ops mailing list
> [email-masked]
> https://www.ietf.org/mailman/listinfo/v6ops

---

From: Mark ZZZ Smith <[email-masked]>
Date: Mon, 1 Oct 2012 13:28:10 -0700 (PDT)

Hi,

----- Original Message -----
> To: [email-masked]
> Sent: Monday, 1 October 2012 8:04 PM

Can you point to an RFC that says that? I don't think there are any statements about where the MTU value used in an RA is allowed or not allowed to come from.

I think what is being proposed is a good use for the RA MTU option. 

> mucking with LAN MTUs.

I think MSS hacking is far worse than what is being proposed. It only works for TCP, not UDP, DCCP, SCTP, GRE, IPsec etc., and also involves far more work per packet when forwarding any packet (Is it TCP? Is it a TCP Syn? Where is the MSS Option (and in IPv6 that could behind a number of extension headers)? Change the MSS value, re-calculate and update the TCP and IP header checksums). Given that these devices are likely to be battery powered, this amount of additional per-packet processing could measurably shorten battery life.

---

From: Mark Andrews <[email-masked]>
Date: Tue, 02 Oct 2012 10:50:35 +1000

In message <[email-masked]>, =?iso-8859-1?q?R
=E9mi_Despr=E9s?= writes:
> >> If you really care about delay, MSS clamping is an option which isn't intr
> usive to the LAN as it affects only connections via the mobile gateway.
> > 

It hurts everyone by not making sites aware that they need to
generate and expect PTB packets.  It hurts everyone because it lets
sites that block all ICMP continue to operate despite the configuration
being broken.  It rewards bad behaviour.

> FYI, something was proposed about this in May, ref. www.ietf.org/mail-archive
> /web/v6ops/current/msg13015.html, namely:
> "In any node, before entering of just after leaving a link whose MTU is the s
> ame in both directions (Ethernet, 6rd, DS-lite, ...), if the MSS advertised i
> n a received TCP SYN packet exceeds the link MTU minus the TCP/IP header leng
> th (40 octets in IPv4, 60 in IPv6), reduce it to this value, and adjust IP an
> d TCP checksums accordingly."
> 
> In principle, I can't any longer afford to be active in v6ops, but still look
>  at some of the exchanged mails and may provide brief comments. Dan Wing is a
> dded as cc because he privately said in May he liked this proposed behavior.
> 
> Regards,
> RD
> 
-- 
Mark Andrews, ISC
1 Seymour St., Dundas Valley, NSW 2117, Australia
PHONE: +61 2 9871 4742                 INTERNET: [email-masked]

---

From: Lorenzo Colitti <[email-masked]>
Date: Tue, 2 Oct 2012 17:00:58 +0900

> everything to 1280 in the first place.

How? By statically configuring the MTU on all the hosts behind the phone
and all the servers on the Internet?

> and the server being connected, but just between the user's client and

Not just between the client and the phone, but in the reverse path as
well. If the MTU of the phone-GGSN link is 1480, then the first >1480
packet that the server sends to the client will cause the GGSN to send a
"packet too big, MTU=1480" message back to the server. That's much more
expensive and subject to loss than the path from the client to the phone.

> intrusive to the LAN as it affects only connections via the mobile

You're suggesting that the network rewrite all the TCP packets emitted by
the hosts behind it, instead of informing said hosts that the MTU to every
other address on the Internet is lower. That doesn't seem to make sense to
me.

> I beg to differ, for reasons of technical correctness, not throughput.

I would argue it's more correct to tell the hosts something ("if you send
packets larger than this MTU, you're going to have a bad time") than to
tell them nothing. The hosts are free to ignore the MTU - its use is only
(lower-case) recommended by RFC 4861.

> notion of LAN link MTU?

That router is is unable to offer bidirectional connectivity to hosts on
the link, because if the hosts send it packets from the mobile carrier
space, they get dropped by the ISP on the other side due to ingress
filtering.

What about endpoints that do ignore MTU received in RAs? (I think I
> observed several that flat-out do ignore MTU)
>

They use PMTUD, which we all agree must work.

> and it's not a handset, but a SOHO backup router.

Well, but it's a soho backup router with a /64 from that can't be routed
but requires the whole environment to be bridged, and that doesn't allow
the network to use the same IPv6 addresses that it uses when the main link
is up. Not a great backup router.

---

From: Rémi Després <[email-masked]>
Date: Tue, 2 Oct 2012 10:17:57 +0200

> being broken.  It rewards bad behaviour.

Different understanding:
(a) Gandma won't be hurt, on the contrary, if her applications that happen to be TCP (without her being aware) work in more circumstances than before. (She couldn't care less about PTB and ICMP!)
(b) Proper PMTUD operation, with PTB propagation everywhere, is still needed for UDP applications, and even for TCP applications if PMTU bottlenecks are remote with no MSS clamping done where they take place. It definitely remains a MUST.

Some prefer to offer as good a service as possible *in the world as is* (including with temporary PMTUD unreliability,  a current fact of life). 
Others, to accelerate worldwide debugging (they think), prefer to purposely maintain for some customers adverse consequences of wrong behaviors for which they aren't responsible. 
Each one remains free to make his own choice.

Regards,
RD