Subject: [tcpm] Inner Space and MTU

From: Bob Briscoe <[email-masked]>
Date: Sun, 16 Nov 2014 19:24:20 +0000

Ted,

Thx for the point you raised about when I presented Inner Space in 
tcpinc. As Martin Stiemerling requested, some issues were not 
specific to tcpinc, so I am moving them to tcpm. When resolved, I 
will send a digest to tcpinc.

Regarding the MTU issue you raised, I don't think it requires a 
redesign, but it is a valid concern. Therefore, I propose to add the 
following to the section "Interaction with the Pre-Existing TCP API"

"Some applications interrogate the TCP stack to determine the path 
max transmission unit (PMTU), e.g. in order to optimize application 
message boundaries within the datastream. From the viewpoint of such 
applications, TCP options subtract the same amount from the PMTU 
whether they are Outer or Inner Options. However, the 8 (or 12) octet 
InSpace header and the alignment padding represent extra overhead. 
Therefore, for such applications, the TCP stack as seem through the 
socket API will seem similar to a tunnel that reduces the useful size 
of the PMTU. This could lead to fragmentation until such applications 
are updated. Nonetheless, most such applications already include code 
to adapt to PMTU reduction by tunnels."

Is this an acceptable resolution?

Bob

________________________________________________________________
Bob Briscoe,                                                  BT

---

From: Yuchung Cheng <[email-masked]>
Date: Mon, 17 Nov 2014 12:30:58 +0800

On Mon, Nov 17, 2014 at 3:24 AM, Bob Briscoe <[email-masked]> wrote:
> Ted,
>
> Thx for the point you raised about when I presented Inner Space in tcpinc.
> As Martin Stiemerling requested, some issues were not specific to tcpinc, so
> I am moving them to tcpm. When resolved, I will send a digest to tcpinc.
>
> Regarding the MTU issue you raised, I don't think it requires a redesign,
> but it is a valid concern. Therefore, I propose to add the following to the
> section "Interaction with the Pre-Existing TCP API"
>
> "Some applications interrogate the TCP stack to determine the path max
> transmission unit (PMTU), e.g. in order to optimize application message
> boundaries within the datastream. From the viewpoint of such applications,
> TCP options subtract the same amount from the PMTU whether they are Outer or
> Inner Options. However, the 8 (or 12) octet InSpace header and the alignment
> padding represent extra overhead. Therefore, for such applications, the TCP
> stack as seem through the socket API will seem similar to a tunnel that
> reduces the useful size of the PMTU. This could lead to fragmentation until
> such applications are updated. Nonetheless, most such applications already
> include code to adapt to PMTU reduction by tunnels.
Reasonable but I'd remove the last sentence.

IMO it's app's own risk doing these kind of optimizations.
The particular fragmentation issue has motivated the Linux auto-corking feature:
http://lwn.net/Articles/576263/

However an interesting question is for implementation that exports the
MSS to the application (e.g., Linux TCP info tcpi*mss fields): should
it include or exclude the inner-space option?

> _______________________________________________
> tcpm mailing list
> [email-masked]
> https://www.ietf.org/mailman/listinfo/tcpm

---

From: "Scharf, Michael (Michael)" <[email-masked]>
Date: Mon, 17 Nov 2014 13:44:50 +0000

> it include or exclude the inner-space option?

Doesn't the same question apply to all variables that refer to byte stream characteristics in bytes? For instance, tcpi_*_ssthresh? (Which is less likely to be used by an app, but one never knows...)

Another interesting question is what a stack with cwnd in bytes does? Does the value include or exclude the inner-space options? What happens if the value is exposed to an app?

Possibly, other shim layer protocols on top of TCP run into similar questions.

Michael

---

From: Joe Touch <[email-masked]>
Date: Mon, 17 Nov 2014 10:05:56 -0800

> it include or exclude the inner-space option?

First, the MSS value used in the header does NOT include TCP or IP
options as per rfc6691

However, note that MSS has meaning at the TCP segment level

The user-level shouldn't be involved in this. Sec 4 of that RFC explains
why - the space used by options can vary (e.g., SACK), so there is no
one appropriate MSS to report to the application.

To the user, TCP is a bytestream. Wanting to expose segment boundaries
to the user (or even implementing it in Linux) doesn't make it possible.

Joe

---

From: Ted Hardie <[email-masked]>
Date: Mon, 17 Nov 2014 11:06:35 -0800

Hi Bob,

Some thougths in-line.

On Sun, Nov 16, 2014 at 8:30 PM, Yuchung Cheng <[email-masked]> wrote:

> tcpinc.
> > such applications are updated.

​
So, the best thing to do from my perspective would be to split IP_MTU
into TCP_MTU and UDP_MTU in getsockopt(); if inner headers would be in
use for TCP, then the MTU returned by the new call would be lower than
that returned for UDP_MTU.  If not, they would be the same.  (You'd need to
figure out what to do if IP_MTU were used, but an obvious choice here
would be to return the lower of the two values if they differ.)

In advance of something like that emerging, the conservative thing to do
would be for the application to do in an Inner Space-approved world would
be to assume it's always in use, and to adjust its message boundaries on
that
basis.

regards,

Ted

> Nonetheless, most such applications already

---

From: Joe Touch <[email-masked]>
Date: Tue, 18 Nov 2014 11:24:21 -0800

> Some thougths in-line.
...
> into TCP_MTU and UDP_MTU in getsockopt();

TCP might report an MTU for diagnostics, but it's a byte-oriented
protocol so this value shouldn't be part of what's exposed to the
userland application per se.

It only gives the incorrect impression that socket boundaries are
correlated to TCP segment boundaries.

Additionally, as I've already noted, TCP MSS doesn't take into account
the option space needed by either IP or TCP. TCP is in a position to use
that value to limit how much application data is added to those headers
AFTER they're formed.

At the application API, there's no meaningful TCP MTU. The space that
remains in a segment depends on the size of the options used *for that
segment*, and those option sizes already vary.

Joe

---

From: Joe Touch <[email-masked]>
Date: Tue, 18 Nov 2014 11:58:15 -0800

On 11/18/2014 11:48 AM, Ted Hardie wrote:
> Hi Joe,
> 
> On Tue, Nov 18, 2014 at 11:24 AM, Joe Touch <[email-masked]
> <mailto:[email-masked]>> wrote:
> 
> 
> 
> ​At the moment, you can query the IP_MTU, and some TCP-using applications
> attempt to use that information to craft messages that fit within the MTU. 

I can play PowerBall too, but that doesn't mean I'll be right (or that
the app will either).

IMO, an app that tries to use MTU when interfacing is doing a lot of
work that TCP basically ignores in at least two ways:

	1) the options, which (as noted) aren't considered and can
	vary (and the user can't predict - users don't know how
	the might be aligned, even if they're stable)

	2) TCP grabs what it needs to fill the next segment
	based on what's available in the window. That window
	need not always indicate a complete MSS; its position
	depends on what has been ACKd of what has been sent

Even with Nagle disabled, there's no way to control that interaction. So
a TCP MTU just helps the app operate under a delusion.

TCP is most efficient when the app hands over data as soon as it can.
Anything else is asking for interference, not potentially helping.

> The IP_MTU may well be a foot gun in its common usage, but it does get
> used. 

That makes more sense for raw socket apps and UDP, but makes no sense at
all for TCP for the same reason.

> If those applications could query the socket to discover what the maximum
> effective TCP payload size was, I believe they would use it instead. 

Sure. I might use it to guess a PowerBall ticket too. That doesn't mean
it's actually useful or will ever be.

I agree that apps can, will, and already do think they can do better.
They're wrong. Let's not play into that delusion.

Joe

---

From: Joe Touch <[email-masked]>
Date: Tue, 18 Nov 2014 13:34:37 -0800

On 11/18/2014 12:51 PM, Ted Hardie wrote:
> ​Applications aren't interested in TCP efficiency in isolation; they
> ​are interested in it in an application context.  Sometimes that
> causes them to make choices different from "hand over the data
> as soon as you can in whatever chunks you have".

When they do, they ought to be using a different transport. It isn't
useful for an app to game its interface to TCP.

...
> ​Do you oppose Bob's language as well?

His first sentence, IMO, only serves to support the delusion that apps
can optimize by managing their interaction with the TCP stack.

However, I'm not too interested in wordsmithing when there are other,
(IMO) more fundamental problems with his approach.

Joe

---

From: Ted Hardie <[email-masked]>
Date: Tue, 18 Nov 2014 11:48:46 -0800

Hi Joe,

If those applications could query the socket to discover what the maximum
effective TCP payload size was, I believe they would use it instead.  That
might
get reported by taking pessimistic estimates of the variable bits plus the
results
of a path mtu discovery.  How it gets reported may be more important than
the details
of how the stack estimates it, though.

Failing that, those applications will try to discern whether inner space
headers
are in use and will do their own estimates of the impact on available
space.  Until
they have been updated to discern that, the 8 (or 12) octet InSpace header
and the
alignment padding will blow them up, as Bob's text notes.  If the working
group wants
to just note the issue and move on, then something along his text is likely
fine.  If
it wants to suggest more, I still believe recognizing the reality and
providing something
better than the current state is worth considering.

Just my two cents,

Ted

---

From: Ted Hardie <[email-masked]>
Date: Tue, 18 Nov 2014 12:51:25 -0800

> I can play PowerBall too, but that doesn't mean I'll be right (or that
​Hmm.  The PowerBall odds are: 1 in 175,223,510 for the Jackpot.  If
those approximate your personal odds of guessing the payload available
to TCP correctly given an MTU, you have one heck of a Path MTU and
some startling variability.  Or possibly you just meant getting your money
back,
which is approximately 1 in 55.  Even that seems pessimistic, since the
actual
problem only occurs if the application's guess is over-generous, resulting
in fragmentation.  But let's take it as read.

> ​It does make more sense for UDP, in general, and it sometimes drives
applications to recreate TCP's other characteristics on top of UDP.​

Ted​

---

From: "Scheffenegger, Richard" <[email-masked]>
Date: Wed, 19 Nov 2014 20:23:08 +0000

Hello Joe,

>> Applications aren't interested in TCP efficiency in isolation; they 
>> are interested in it in an application context.  Sometimes that 
>> causes them to make choices different from "hand over the data as soon 
>> as you can in whatever chunks you have".
>
> When they do, they ought to be using a different transport. 

I tend to agree if we'd be living in a perfect world...

> It isn't useful for an app to game its interface to TCP.

au contraire! In my context (storage) there is a very well-known protocol that switched from UDP to TCP (at a time when SCTP wasn't available; one can argue, it still isn't in most enterprises...). In order to avoid excessive head-of-line blocking, it does make sense for an app to feed the sending socket the data in small chunks, to allow more critical data to get processed (that is, transported, dealt with on the server, and responded to) faster. And that particular piece of behavior is actually implemented in one particular user-space NFS implementation, fairly recently (2011/12 IIRC).

Or, to use a (in-)famous word, to circumvent bufferbloat a little bit at least (tcp send buffer).

Also, even though NFS does not explicitly maintain segment boundaries (or cares about TCP MSS), the frequent use of PSH serves to align the RPC/NFS calls to the extent that many implementations fast-path NFS processing by expecting fixed (tcp data) offsets...

Ideally, NFS (or rather, RPC) would be using SCTP, to address the transport protocol demands, no doubt.

In any case, I'm not interested in discussing this to extinction ;)

Richard Scheffenegger

---

From: Joe Touch <[email-masked]>
Date: Wed, 19 Nov 2014 12:35:55 -0800

On 11/19/2014 12:23 PM, Scheffenegger, Richard wrote:
> au contraire! In my context (storage) there is a very well-known
> protocol that switched from UDP to TCP (at a time when SCTP wasn't
> available; one can argue, it still isn't in most enterprises...). In
> order to avoid excessive head-of-line blocking, it does make sense for
> an app to feed the sending socket the data in small chunks, to allow
> more critical data to get processed (that is, transported, dealt with on
> the server, and responded to) faster. And that particular piece of
> behavior is actually implemented in one particular user-space NFS
> implementation, fairly recently (2011/12 IIRC).

To be clear - a given implementation can be gamed all day long.

Once the OS or protocol is updated, though, *all bets are off*, and all
of those optimizations can - and will - come back to bite the programmer.

> at least (tcp send buffer).

Which comes back and bites you once AQM is deployed.

> Also, even though NFS does not explicitly maintain segment
> boundaries (or cares about TCP MSS), the frequent use of PSH serves to align the
> RPC/NFS calls to the extent that many implementations fast-path NFS
> processing by expecting fixed (tcp data) offsets...

Note that PSH is part of the TCP API.

Joe