Subject: [tcpm] Sending data before ACK(SYN) receipt

From: William Allen Simpson <[email-masked]>
Date: Fri, 13 May 2011 10:27:11 -0400

This arose on the NANOG list, and I've not been able to figure out
how it happens.  It does happen consistently, and seems relevant to
this list.

Linux 2.6.18 on the server side sometimes sends:

# sh capture debug-in
8 packets captured
   1: 21:49:13.461554 client.32929 > server.3306: S
4107544000:4107544000(0) win 65535 <mss 1380,nop,wscale 3,sackOK,timestamp
2065216038 0>
   2: 21:49:13.462073 server.3306 > client.32929: S
2601320299:2601320299(0) ack 4107544001 win 5792 <mss 1460,sackOK,timestamp
2581054349 2065216038,nop,wscale 7>
   3: 21:49:13.462210 server.3306 > client.32929: P
2601320300:2601320363(63) ack 4107544001 win 46 <nop,nop,timestamp
2581054349 2065216038>
   4: 21:49:13.519061 client.32929 > server.3306: . ack 2601320300
win 8208 <nop,nop,timestamp 2065216096 2581054349>

That is, the server sends data before receiving ACK(SYN) to complete the
three-way handshake.  Cisco firewalls drop segment 3, as a violation of
the "tcp-3whs-failed" rule.

   5: 21:49:14.135384 client.32929 > server.3306: P
4107544001:4107544003(2) ack 2601320300 win 8208 <nop,nop,timestamp
2065216712 2581054349>
   6: 21:49:14.135521 server.3306 > client.32929: . ack 4107544003
win 46 <nop,nop,timestamp 2581055023 2065216712>
   7: 21:49:16.461981 server.3306 > client.32929: P
2601320300:2601320363(63) ack 4107544003 win 46 <nop,nop,timestamp
2581057349 2065216712>
   8: 21:49:16.618147 client.32929 > server.3306: . ack 2601320363
win 8208 <nop,nop,timestamp 2065219195 2581057349>

The good news is that it doesn't appear to be fatal, but stalls the
connection for 2+ seconds until retransmit.  Egregiously contrary to
intent of speeding up the connection.

Anybody know how this was triggered?  I cannot find any documentation
about this in the Linux stack.  I've tried comparing sysctls to current
values, nothing obvious.

Also, the server window and wscale seem a bit flaky, too....

---

From: Joe Touch <[email-masked]>
Date: Fri, 13 May 2011 08:57:50 -0700

Hi, all,

On 5/13/2011 7:27 AM, William Allen Simpson wrote:
> three-way handshake.

That seems like the server is the party violating TCP. SEND calls in the 
SYN-RECEIVED state are supposed to:

"Queue the data for transmission after entering ESTABLISHED state."

The server is jumping the gun sending the data early when it is in the 
SYN-RECEIVED state.

 > Cisco firewalls drop segment 3, as a violation of

Perhaps. It's consistent with enforcing what the server should be doing, 
from RFC793 processing rules, though. I.e., it seems compliant, though 
not desired.

Joe

---

From: Tim Shepard <[email-masked]>
Date: Fri, 13 May 2011 13:48:43 -0400

I've always understood that TCP (the protocol) allows data to travel
with the SYN, and that the only important rule was that the TCP should
not deliver any data up to a naive application without having first
received an ACK of its own SYN (proving return routability, not a
stale packet, etc).

I remember watching Phil Karn's KA9Q NOS exchanging packets and seeing
it do a full SMTP exchange with remarkably few packets, and I think it
included data with the SYN in both directions.  This was all good
stuff.  (It was doing the best it could do in terms of packet and
round-trip efficiency and getting the transfer-of-responsibility for
the e-mail message right, short of solving the Two Generals' Problem.)

A non-naive application that got to see and examine the data with the
SYN and produce some data that could be included with the SYN-ACK
packet could be a good thing.  We need to be careful not to rule out
this sort of optimization.  (The author of such a non-naive
application has to understand the different nature of this data
received before an ACK of its own TCP's SYN has been received, and
take on some extra responsibility for coping with exceptional
situations.)

Round trip delays are one of the most expensive thing we've got.
Avoiding them can be important to performance, and if including data
with the SYN helps, well, that's good.

			-Tim Shepard
			 [email-masked]

---

From: Joe Touch <[email-masked]>
Date: Fri, 13 May 2011 10:57:12 -0700

On 5/13/2011 10:48 AM, Tim Shepard wrote:
> with the SYN,

The issue is whether it allows separate data packets after the SYN but 
before the SYN-ACK. There doesn't appear to be a rule against receiving 
such packets, but the description clearly rules out generating them, AFAICT.

Joe

---

From: "Borman, David" <[email-masked]>
Date: Fri, 13 May 2011 18:56:19 +0000

> Linux 2.6.18 on the server side sometimes sends:
Is this the kernel.org version, or one modified by some distro?

> 4107544000:4107544000(0) win 65535 <mss 1380,nop,wscale 3,sackOK,timestamp

Just on the face of it from the packet traces, assuming this is intentional, it looks like the server is being opportunistic in sending the data, assuming that by the time the data packet gets there, the client will have already processed the SYN/ACK and sent back an ACK completing the 3 way handshake, transitioning to ESTABLISHED state, and thus the data packet would appear to be perfectly valid and accepted.  Thus the server eliminates one RTT in starting up the connection.

Of course the problem is, as in this case, the server has no way of knowing if the data packet was dropped, and if so, causes the connection to timeout to retransmit the data.

Of course, the danger here is if the initial SYN is bogus, the server will have already sent out data before verifying connection establishment.

This would be no different than a scenario where a TCP opportunistically sent data beyond the advertised window, on the assumption that by the time the data arrived at the other side, it will have processed the previous data and advanced the window and be able to accept the additional data.  From the client side everything looks normal, other than, that last new data showed up really fast after the window was advanced.  And again, if the window *doesn't* advance, the opportunistic data will be dropped, potentially leading to a timeout on the server, slowing down the connection instead of speeding it up.

> Also, the server window and wscale seem a bit flaky, too....

I don't think so.  A default wscale=7 is reasonable, and in the data packet the server scaled the 5792 window from the SYN packet to 46 (5792/(2^7)).  The initial window of 5792 seems to be roughly 4 ethernet packets: (1500-52)*4 = 5792

			-David Borman

> _______________________________________________
> tcpm mailing list
> [email-masked]
> https://www.ietf.org/mailman/listinfo/tcpm

---

From: "Anantha Ramaiah (ananth)" <[email-masked]>
Date: Fri, 13 May 2011 12:05:51 -0700

<snip>

> from RFC793 processing rules, though. I.e., it seems compliant, though

It could be a bogus SYN, which causes server to send SYN+ACK to the
client which would solicit an RST. Typically motive of the firewall is
to prevent such things from happening, hence it would drop the data
segment since the connection state is not ESTAB until the ACK receipt. 

lets not get into an argument about middleboxes are evil etc., but just
outlining the common practice :-) Also lets not get too much carried
away with the terms server/client, a firewall's intention is to protect
the box that is behind it, it could be anything.

-Anantha

---

From: "Smith, Donald" <[email-masked]>
Date: Fri, 13 May 2011 13:41:41 -0600

Sharing: Author's permission required.
[email-masked]

> -----Original Message-----
> From: [email-masked] [mailto:[email-masked]] On Behalf Of
> Anantha Ramaiah (ananth)
> Sent: Friday, May 13, 2011 1:06 PM
> To: Joe Touch; William Allen Simpson
> Cc: TCP Modifications WG
> Subject: Re: [tcpm] Sending data before ACK(SYN) receipt
>
> <snip>
>
Agreed
"Several examples of connection initiation follow.  Although these
  examples do not show connection synchronization using data-carrying
  segments, this is perfectly legitimate, so long as the receiving TCP
  doesn't deliver the data to the user until it is clear the data is
  valid (i.e., the data must be buffered at the receiver until the
  connection reaches the ESTABLISHED state)."

Servers that exibit this behavior could be used in a spoofed source address syn flood with reflection and amplification depending on how much data is sent in the syn/ack response. So I would advise against it:)

> client which would solicit an RST. Typically motive of the firewall is
I believe your "bogus" syn is a spoofed source address syn correct?

> the box that is behind it, it could be anything.

This communication is the property of Qwest and may contain confidential or
privileged information. Unauthorized use of this communication is strictly
prohibited and may be unlawful.  If you have received this communication
in error, please immediately notify the sender by reply e-mail and destroy
all copies of the communication and any attachments.

---

From: William Allen Simpson <[email-masked]>
Date: Fri, 13 May 2011 16:50:21 -0400

> received an ACK of its own SYN (proving return routability, not a
We're in violent agreement.  See RFC-6013.

But that's not what's going on here.  This is a packet between the server
SYNACK and client ACK(SYN), more relevant to our discussions of
draft-cheng-tcpm-fastopen and draft-simpson-tcpct-rr.

> stuff....
As I'm sure you remember, I was also an avid TCP-GROUP user and
developer for KA9Q (Net and later NOS).  :-)

---

From: William Allen Simpson <[email-masked]>
Date: Fri, 13 May 2011 17:08:24 -0400

> Is this the kernel.org version, or one modified by some distro?
Red Hat 2.6.18-164.9.1.el5.  Not my machine, so I'm unsure.  They've not
found any modifications in usr/src, so I'm guessing it's plain vanilla.
Anybody here from Red Hat or LKML?

>> # sh capture debug-in
>> 8 packets captured
>>   1: 21:49:13.461554 client.32929>  server.3306: S
>> 4107544000:4107544000(0) win 65535<mss 1380,nop,wscale 3,sackOK,timestamp
>> 2065216038 0>
>>   2: 21:49:13.462073 server.3306>  client.32929: S
>> 2601320299:2601320299(0) ack 4107544001 win 5792<mss 1460,sackOK,timestamp
>>   3: 21:49:13.462210 server.3306>  client.32929: P
>> 2601320300:2601320363(63) ack 4107544001 win 46<nop,nop,timestamp
>> 2581054349 2065216038>
>>   4: 21:49:13.519061 client.32929>  server.3306: . ack 2601320300
>> win 8208<nop,nop,timestamp 2065216096 2581054349>
>>
>>   5: 21:49:14.135384 client.32929>  server.3306: P
>> 4107544001:4107544003(2) ack 2601320300 win 8208<nop,nop,timestamp
>> 2065216712 2581054349>
>>   6: 21:49:14.135521 server.3306>  client.32929: . ack 4107544003
>> win 46<nop,nop,timestamp 2581055023 2065216712>
>>   7: 21:49:16.461981 server.3306>  client.32929: P
>> 2601320300:2601320363(63) ack 4107544003 win 46<nop,nop,timestamp
>> 2581057349 2065216712>
>>   8: 21:49:16.618147 client.32929>  server.3306: . ack 2601320363
>> win 8208<nop,nop,timestamp 2065219195 2581057349>
>>
Yes.  But I've never seen this in my Ubuntu to *BSD testing, so my question is
how is this triggered?  Reportedly, this machine has no problems communicating
in-house, only across the Internet.  Maybe it's always on for the distro???

> Of course, the danger here is if the initial SYN is bogus, the server will have already sent out data before verifying connection establishment.
I agree there are security holes.  I'm reporting, not advocating.  But it
negates any possibility of our related drafts working correctly in the field.

> This would be no different than a scenario where a TCP opportunistically sent data beyond the advertised window, on the assumption that by the time the data arrived at the other side, it will have…
True.  As observed!

> I don't think so.  A default wscale=7 is reasonable, and in the data packet the server scaled the 5792 window from the SYN packet to 46 (5792/(2^7)).  The initial window of 5792 seems to be roughly…
Aha, I miscalculated.  Thanks.

---

From: Joe Touch <[email-masked]>
Date: Fri, 13 May 2011 14:08:30 -0700

> draft-cheng-tcpm-fastopen and draft-simpson-tcpct-rr.

As others have pointed out, if the SYNACK arrives at the client *and* is 
ACK'd, then the client would be in ESTABLISHED and the a data would be 
accepted.

I.e., the server behavior expects that this happens at the client. If 
not - e.g., if the SYNACK is lost - they you get a stall.

That's normal behavior.

However, the TCP spec says that the server shouldn't have been sending 
this data yet. Until it gets the final ACK, the only data should have 
been in the SYN.

So the firewall is compliant in dropping the data packet - as would be 
the client, if it did so. The result in that case is also a stall.

I.e., this isn't about data in the SYN; it's about when each side is 
permitted to start sending data after the SYN.

Joe

---

From: Tim Shepard <[email-masked]>
Date: Fri, 13 May 2011 17:10:08 -0400

> such packets, but the description clearly rules out generating them, AFAICT.

Ah, I completely misunderstood what this is about.

Now I've looked more closely at Bill's original message and I see the
interesting packet...

That is weird and interesting.

I peeked at mysql-5.1-5.1.49/server-tools/instance-manager/listener.cc
which appears to be the relevant user-mode source code (judging from
the server port number) and I see it is doing this:

  socket(AT_INET, SOCK,STREAM, 0)

  int arg=1;
  setsockopt( , SOL_SOCKET, SO_REUSEADDR,  &arg, sizeof arg)

  listen( , )

  int flags = fcntl( , F_GETFL, 0)
  fcntl( , F_SETFL, flags | O_NONBLOCK);

  int flags= fcntl( , F_GETFD, 0);
  fcntl( , F_SETFD, flags | FD_CLOEXEC);

  select(n , &reads, 0, 0, &tv);

  accept( , 0 , 0)

and I would think that the accept call would not return until the
connection gets out of SYN_RCVD state.

Weird.

Did linux really do this?

I'm wondering is there another middlebox involved that sent an ACK
back to the server?

(Bill, did you collect that trace on the server itself?)

I just tried to reproduce this on a linux server that I have access
to that already has mysql server listening on 3306, and I do not see
this weird behavior (kernel is 2.6.32-5-amd64 from Debian).  I did
"telnet server 3306" from home to trigger the connection, and
collected this trace on the server:

16:32:17.931509 IP client.52527 > server.3306: Flags [S], seq 1071911238, win 5840, options [mss 1460,sackOK,TS val 851353709 ecr 0,nop,wscale 7], length 0
16:32:17.931539 IP server.3306 > client.52527: Flags [S.], seq 1870570388, ack 1071911239, win 5792, options [mss 1460,sackOK,TS val 4291664564 ecr 851353709,nop,wscale 7], length 0
16:32:17.969949 IP client.52527 > server.3306: Flags [.], ack 1, win 46, options [nop,nop,TS val 851353719 ecr 4291664564], length 0
16:32:17.988568 IP server.3306 > client.52527: Flags [P.], seq 1:63, ack 1, win 46, options [nop,nop,TS val 4291664578 ecr 851353719], length 62
16:32:18.023935 IP client.52527 > server.3306: Flags [.], ack 63, win 46, options [nop,nop,TS val 851353733 ecr 4291664578], length 0
16:32:28.008378 IP server.3306 > client.52527: Flags [F.], seq 63, ack 1, win 46, options [nop,nop,TS val 4291667083 ecr 851353733], length 0
16:32:28.044627 IP client.52527 > server.3306: Flags [F.], seq 1, ack 64, win 46, options [nop,nop,TS val 851356238 ecr 4291667083], length 0
16:32:28.044647 IP server.3306 > client.52527: Flags [.], ack 2, win 46, options [nop,nop,TS val 4291667092 ecr 851356238], length 0

Note in my trace the ack of the server's SYN arrives before the
server sends those first 63 bytes.

I tried about a dozen more times, and never saw that data from the
server go out before the ack from the client.

Has anyone else managed to reproduce the behavior shown in the trace
Bill posted?

			-Tim Shepard
			 [email-masked]

---

From: William Allen Simpson <[email-masked]>
Date: Tue, 17 May 2011 10:00:54 -0400

> Has anyone else managed to reproduce the behavior shown in the trace
Neither Tim nor I can reproduce this on current debian or Ubuntu.

This seems to be a new and undocumented feature of Red Hat Enterprise
Linux 5 released in an update late last year.  Or at least that's the
current best guess.

Are there any Red Hat folks here?

---

From: <[email-masked]>
Date: Tue, 17 May 2011 15:08:52 +0100

Between the oddities in redhat's gcc variant and its kernel, is it even worth tracking down red hat problems? 

> From: [email-masked] [mailto:[email-masked]] On 
> Behalf Of William Allen Simpson
> Sent: 17 May 2011 15:01
> To: [email-masked]
> This seems to be a new and undocumented feature of Red Hat 
> Enterprise Linux 5 released in an update late last year.  Or 
> at least that's the current best guess.
>

---

From: Carsten Bormann <[email-masked]>
Date: Tue, 17 May 2011 17:16:54 +0200

On May 13, 2011, at 23:08, Joe Touch wrote:

> However, the TCP spec says that the server shouldn't have been sending this data yet. Until it gets the final ACK, the only data should have been in the SYN.

It is not clear to me that the text in 793 isn't a classical case of overspecification.

The *protocol* typically does not care -- the connecting peer will have sent the final ACK before it gets the data, so it cannot distinguish the deviant behavior from the specified one (unless it has a special delay built in here).

The *service* (distributed abstract machine leaky abstraction...) indeed does not want data handed down early to be leaking out the wire before the three-way handshake completes.  But if the service user doesn't care about that leak?  Then this is a valid optimization.  (Modulo the amplification aspect, but I'm not sure this is really that relevant with 63? bytes.)  So a high-quality specification should be warning about the effects of sending early (as in SHOULD NOT unless...), but not right outlaw it (MUST NOT).

It is not really the business of the middlebox to second-guess the desires of the host.  
(But of course we know many middleboxes are built to do exactly that.)
(And, of course, one way for the server to maintain the RTT optimization in the presence of pesky second-guessing middleboxes is to send the data *twice*, once immediately after the SYN-ACK and once the handshake-completing ACK arrives.)

What I'm trying to say here is that deviant behavior that is indistinguishable from compliant behavior is interoperable*).  The interesting question is never whether you are compliant, but whether you are interoperable.  The bad guy is not the server implementing an interoperable optimization, but the middlebox preventing the interoperable optimization under the pretext of non-compliance.

(Sorry for bringing up these philosophical points here, which mostly are of import when writing new protocol specifications, not so much when maintaining widely implemented ones.)

Gruesse, Carsten

*) ... and the existence of such behavior is a diagnostic for overspecification.

---

From: Joe Touch <[email-masked]>
Date: Tue, 17 May 2011 09:21:40 -0700

Hi, Carsten,

> It is not clear to me that the text in 793 isn't a classical case of

I do think there is a goal - of limiting the amount of data held until 
the 3WHS completes.

> The *protocol* typically does not care -- the connecting peer will
> have sent the final ACK before it gets the data, so it cannot
> distinguish the deviant behavior from the specified one (unless it
> has a special delay built in here).

First, that assumes no packets are lost. If the SYN-ACK is lost, or the 
final ACK is lost, then the behavior is different (and hiccups).

> The *service* (distributed abstract machine leaky abstraction...)
> indeed does not want data handed down early to be leaking out the
> wire before the three-way handshake completes.

See above for reasons why that affects the protocol - or at least its 
implementation.

> But if the service
> user doesn't care about that leak?  Then this is a valid
> optimization.

Right, but so is having a middlebox decide to delete that advance data 
to prevent the server from being overloaded until the connection completes.

So one person's optimization is another's overload. Then that person's 
protection is the first person's problem (creating a stall).

 > (Modulo the amplification aspect, but I'm not sure
> this is really that relevant with 63? bytes.)  So a high-quality
> specification should be warning about the effects of sending early
> (as in SHOULD NOT unless...), but not right outlaw it (MUST NOT).

The current focus is to be conservative - unsurprisingly. The danger of 
being aggressive is that when you fail, it costs more.

> It is not really the business of the middlebox to second-guess the desires of the host.

Yes, unless the middlebox is working on behalf of the host (e.g., under 
control of the admin, etc.). And agreed that some midbox managers 
*think* they're working on behalf of the host even when they're not...

> What I'm trying to say here is that deviant behavior that is indistinguishable from compliant behavior is interoperable*).

Yes, but so is the packet loss created by the midbox ;-) IP drops packets...

>   The bad guy is not the server implementing an interoperable optimization,

Who's 'bad' here - as in many cases - depends on what you're optimizing. 
The fact that the server being aggressive sometimes shoots itself in the 
foot is, IMO, just desserts.

Joe

---

From: William Allen Simpson <[email-masked]>
Date: Wed, 18 May 2011 06:50:16 -0400

On 5/17/11 12:21 PM, Joe Touch wrote:
>> ...
> Yes, unless the middlebox is working on behalf of the host (e.g., under control of the admin, etc.). And agreed that some midbox managers *think* they're working on behalf of the host even when they're not...
>
In this case, the middlebox firewall is under the control of the same
operator as the host, but the middlebox doesn't have a way of turning off
its behavior.  The middlebox *programmers* think they're working on
behalf of the host even when they're not....

> Yes, but so is the packet loss created by the midbox ;-) IP drops packets...
True, I think Dave Borman stated that pretty well earlier.

> Who's 'bad' here - as in many cases - depends on what you're optimizing. The fact that the server being aggressive sometimes shoots itself in the foot is, IMO, just desserts.
>
IMnsHO, I think both are "bad" here.  We've had the Google draft advocating
this early data, and my counter-draft.  But *we* understand that a change to
the TCP protocol needs to be negotiated securely as an option.

Red Hat (apparently) "just did it" without any negotiation.  That's bad.

Cisco shipped a firewall without the ability to turn off the behavior.
That's bad, too.

Thank goodness the operator brought the problem to NANOG!  (And why we
specify these as Experimental drafts, because TCP needs widespread
testing against more than the usual consumer middleboxen.)

I'll add the following to my draft:

Applicability                                                            +

    This specification is intended for network paths under the complete   +
    control of an operator, such as secure tunnels or intra-campus        +
    private links.  Widely deployed security firewalls block the          +
    transmission of these additional data segments, and are outside the   +
    scope of this specification.                                          +