Subject: [tcpm] Really odd WSCALE values

From: Matt Mathis <[email-masked]>
Date: Wed, 25 May 2011 07:06:59 -0400

Measurement Lab ( http://www.measurementlab.net/ ) public data shows a
really odd distribution of receive WSCALE values (below) ranging all
the way up to 14.   The data shows the number of client IP addresses
that invoked a test to some Measurement Lab node, with the given value
the maximum WSCALE value.  NATs and other shared client IP addresses
only count as the largest winscale seen.  -1 indicates not negotiated,
as opposed to negotiated 0.  The data is for the month of 2011
February.

WSCALE values of 9 and above are absurd.  For example at WSCALE of 10,
the maximum rwin value is 64MB and it is quantized in 1kB increments,
so an Ethernet frame is only 1 or 2 clicks.  Above 10, not even one
click per MSS....

Why would somebody use such large scale values?    The population is
too large for it to be errant manual tuning...

Furthermore, this probably tickles a bug in 1323  (you can't avoid
retracting the receiver window, if the flow is rwin controlled and the
read buffer is smaller than the rwin quanta).  However, It is hard to
imagine this being some sort of exploit.   Besides, why would it be
such a large number of distinct client addresses?

Anybody have any ideas about what might be going on?

  WinScaleRcvd   number_of_tests

|          14 |            1568 |
|          13 |               9 |
|          12 |             484 |
|          11 |             722 |
|          10 |            2615 |
|           9 |            1273 |
|           8 |         2042885 |
|           7 |           56212 |
|           6 |           18036 |
|           5 |           16153 |
|           4 |           46208 |
|           3 |          470349 |
|           2 |         3147658 |
|           1 |          250646 |
|           0 |          316808 |
|          -1 |         5271154 |

Any ideas?
Thanks,
--MM--
The best way to predict the future is to create it.  - Alan Kay

---

From: Tim Shepard <[email-masked]>
Date: Wed, 25 May 2011 09:00:23 -0400

> too large for it to be errant manual tuning...

I expect that the 0.04 percent of the population in your data that had
wscale values larger than 9 happened because of manual tuning.  It has
been well-enough known that sometimes on linux in order to saturate a
link with a significant delay-bandwidth product that you need to turn
up {r,w}mem_{max,default} in /proc/sys/net/{ipv4,core} to larger
values, and I believe that wscale is selected so that the 16-bit
window field can fully reflect the amount of buffer space available.
(But it has been many years since I've found I need to do that sort of
tuning by hand.)

> so an Ethernet frame is only 1 or 2 clicks.  Above 10, not even one

I don't understand what the problem is with large wscale values.
When wscale is large, you don't have to retract window as you move
through the sequence space, as long as the window you've advertised is
less than or equal to the window you have (the difference being the
low-order bits that don't get carried in the 16-bit window field).
When the buffer space allocated to the connection is larger, this
small difference will not matter.

			-Tim Shepard
			 [email-masked]

---

From: rick jones <[email-masked]>
Date: Wed, 25 May 2011 08:13:52 -0700

> (But it has been many years since I've found I need to do that sort of

Having just seen, in another mailing list, an example of someone  
setting [rw]mem_max to 50 million bytes for some of their testing, and  
not "re-tweaking" it later when their test conditions changed, ("TCP  
will deal with it and do the right thing.") I am inclined to agree  
with Tim about the likelihood of it being manual tuning.  Perhaps  
something along the lines of "Ok, well does it get any better if I go  
balls-out on the window size?"

Although, if there were a way to try to "finger print" the ostensibly  
offending IPs that might be goodness... individuals doing slightly  
screwy things can in fact have employers shipping kit.   Did these IPs  
ever actually advertise a window out to the limit enabled by those  
window scaling values?

Matt originally said
> so an Ethernet frame is only 1 or 2 clicks.  Above 10, not even one

Window updates are "supposed" to be non-trivial fractions of the  
window, or at least 2*MSS anyway no?

One other thought - one can almost always advertise a smaller window  
regardless of WSCALE (yes, modulo the granularity imposed), but once  
the WSCALE is set that fixes an upper bound one cannot exceed.  In  
that sense, one might have a school of thought that believes that  
larger values of WSCALE are better than smaller ones - for the  
flexibility they provide to say advertise more window when one  
discovers the bandwidth delay product is large.   Perhaps an  
application layer version of the (in)famous Linux autotuning.

rick jones
Wisdom teeth are impacted, people are affected by the effects of events

---

From: David Borman <[email-masked]>
Date: Wed, 25 May 2011 11:38:08 -0500

> so an Ethernet frame is only 1 or 2 clicks.  Above 10, not even one

I think that just having the distribution of WSCALE values is not that useful.  What matters is what is the size of the window that is being offered for a given WSCALE value.  As long as the window exceeds the WSCALE by a reasonable amount, things should be fine.  If it doesn't, well, the TCP should already deal with rounding up, even for small WSCALE values.  But yes, if the actual available window is small enough that the scaled offered window can only be 0 or 1, then that might not be the best configuration, but things should still work. :-)

			-David Borman
 
> _______________________________________________
> tcpm mailing list
> [email-masked]
> https://www.ietf.org/mailman/listinfo/tcpm

---

From: Matt Mathis <[email-masked]>
Date: Thu, 26 May 2011 10:36:18 -0400

You may be correct about errant manual tuning.  I will have to think about it.

> small difference will not matter.

My paraphrasing of the problem was not quite correct.   Try 2:

Define wextent to the sequence number of the left edge of the receiver window:
wextent = seg.ack+(2^wscale)*seg.win
Suppose three conditions:
1) wscale too big, in this case I will assume 1 just to make the point
2) the receiver is not making progress.  e.g. it wants to announce a
constant wextent
3) the sender is sending data in smaller increments than (2^wscale).
e.g. 1 byte at a time, and waiting for each ack
In this case you will see an alternating pattern:
seg.ack advances by 1 causing wextent to advance by 1
seg.ack advances by 1, seg.win retracts by 1, causing wextent to retract by 1
There is nothing else that the receiver can do.......

Note that these conditions are quite easy to trigger momentarily with
any piplined bidirectional protocol (e.g. http 1.1) on a busy server,
if there are requests that are smaller than the wscale increment.

So MUST NOT retract the receiver window is flat out incorrect and not possible.

I went back and looked, and RFC 793 & 1122 actually say reasonable
things (SHOULDs in the right places, etc) but  just a bit vague.  I
have a vague recollection of a spec someplace that strongly forbids
retracting the window at all, but I could not find it.  Anybody
suggest other docs that might say such a thing?

Note that retracting the window at all also causes an ambiguity in the
definition of zero window, and if the very last byte is in or out of
window, for example at a firewall.  (If TCP is not doing Nagle, the
last data segment can extend beyond wextent of an ACK that it
crosses).

Dave, We had a conversation about this problem in the context of
1323bis a couple of years ago, but I lost track of where it went.
Since 1323 is the normative definition for wscale, it would at least
be a reasonable place to discuss this problem...

Thanks,
--MM--

---

From: rick jones <[email-masked]>
Date: Thu, 26 May 2011 08:40:27 -0700

> suggest other docs that might say such a thing?

I have similarly vague recollections about retracting the window being  
considered poor form.  My dimm memory thinks the reason was it  
triggered some sort of bug in one of the common stacks at the time.   
What I don't recall was if the issue was in retracting the window at  
all, or if it was in retracting the window beyond what the sender had  
already sent.

A *very* cursory search using "retract tcp window bug" finds this:
http://www.tcpipprotocols.info/bsd-4-3-terminates-zero-window-connection#comments

where someone mentions a possible BSD 4.2 bug.

rick jones