Subject: Re: [tcpm] WGLC: TCP's Reaction to Soft Errors

From: Saikat Guha <[email-masked]>
Date: Wed, 02 May 2007 08:45:23 -0400

On Tue, 2007-04-24 at 23:46 -0400, Mark Allman wrote:
> >         Title           : TCP's Reaction to Soft Errors
> > 	Author(s)       : F. Gont
> > 	Filename        : draft-ietf-tcpm-tcp-soft-errors-05.txt

I read the draft, and to the extent that the draft is informational and
does not standardize treating soft-errors as hard I have no objections.

The behavior described by the draft (that of considering soft errors as
hard during SYN-SENT state after 0/n retransmissions), however, was a
cause of concern for the NAT TCP Behavior document
(draft-ietf-behave-tcp-07.txt), which is currently in the RFC editors
queue. In that document, NATs are basically recommended to silently
delay any ICMP errors by 6 seconds to avoid triggering any of the
mechanism proposed in this document (proviso endpoints can coordinate to
exchange SYN packets through that 6 second hole).

IMHO it would help to point out in Section 3 (Problems that may arise
from TCP's reaction to soft errors) and Section 6.2 (Possible drawbacks:
Deterministic transient network failures) that stacks that treat ICMP
soft errors as hard during connection initiation (at least in the first
6 seconds) make it difficult for applications attempting to initiate TCP
Simultaneous-Open through a BEHAVE compliant NAT.

cheers,
-- 
Saikat
_______________________________________________
tcpm mailing list
[email-masked]
https://www1.ietf.org/mailman/listinfo/tcpm

---

From: Fernando Gont <[email-masked]>
Date: Wed, 02 May 2007 14:07:21 -0300

>Simultaneous-Open through a BEHAVE compliant NAT.

Could you clarify what is the reason for the 6-second delay?

And, when you sent the ICMP error message after the 6 seconds, how 
are TCP stacks expected to react? (why do you send an ICMP error 
instead of a, say, RST segment?)

Kindest regards,

-- 
Fernando Gont
e-mail: [email-masked] || [email-masked]
PGP Fingerprint: 7809 84F5 322E 45C7 F1C9 3945 96EE A9EF D076 FFF1

---

From: Joe Touch <[email-masked]>
Date: Wed, 02 May 2007 12:18:22 -0700

> Could you clarify what is the reason for the 6-second delay?

I helped with this one in BEHAVE...

It allows systems trying to allow two hosts behind NATs to handshake via
SYNs. The outgoing SYNs are there to open the port in the reverse
direction; if they aren't very tightly synchronized, one will show up
coming into a NAT (from the public side) before its pair has a chance to
open that port as it exits that NAT (from the private side). The delay
allows the handshake to occur without ICMP processing shutting it down.

It's valid because there's no timeliness requirement in ICMP responses
(if there were, we'd have big problems everywhere).

> And, when you sent the ICMP error message after the 6 seconds, how are
> TCP stacks expected to react? (why do you send an ICMP error instead of
> a, say, RST segment?)

They ought to react the same way they do now. :-)

You send that error because you got a SYN for which the port wasn't yet
opened, which is a policy violation not a TCP connection failure.

Joe

-- 
----------------------------------------
Joe Touch
Sr. Network Engineer, USAF TSAT Space Segment

---

From: Fernando Gont <[email-masked]>
Date: Wed, 02 May 2007 17:30:22 -0300

>allows the handshake to occur without ICMP processing shutting it down.

Okay. But then, we're safe here... I mean, even if "soft errors" is 
implemented, the simultaneous open will succeed through the NAT (with 
the stuff you proposed)

>(if there were, we'd have big problems everywhere).

If you have any specific concerns of the problems that would arise as 
a result of this, I'd like to know. (not to raise a discussion, but 
rather because I'm wondering what you are thinking of)

>They ought to react the same way they do now. :-)

abort the connection at once, abort it after 75 seconds (or so), or 
abort it only after a user timeout? :-) (all three are currently implemented)

>opened, which is a policy violation not a TCP connection failure.

Do you recall which type/code they send?

Kindest regards,

---

From: Joe Touch <[email-masked]>
Date: Wed, 02 May 2007 13:56:59 -0700

> At 04:18 p.m. 02/05/2007, Joe Touch wrote:
...
> abort it only after a user timeout? :-) (all three are currently

The ICMP would not get sent if the second SYN gets through; that aborts
the pending ICMP which is held for 6 seconds. I.e., if the connection
succeeds there is no ICMP to decipher. If the connection fails, you get
an ICMP error to a pending SYN, which should only abort a pending
connection (and should do so immediately, since it should be interpreted
as if a RST anyway).

> Do you recall which type/code they send?

   REQ-4:  A NAT MUST NOT respond to an unsolicited inbound SYN packet
      for at least 6 seconds after the packet is received.  If during
      this interval the NAT receives and translates an outbound SYN for
      the connection the NAT MUST silently drop the original unsolicited
      inbound SYN packet.  Otherwise the NAT SHOULD send an ICMP Port
      Unreachable error (Type 3, Code 3) for the original SYN, unless
      REQ-4a applies.

i.e., port unreachable, which are supposed to be hard errors. I don't
think this affects either of your docs (soft-errors or ICMP-attacks),
since the ICMP is issued only if the connection has yet to be
established (not covered in the latter) and it's a hard error (not
covered by the former).

Joe

---

From: Joe Touch <[email-masked]>
Date: Wed, 02 May 2007 14:04:06 -0700

> implemented, the simultaneous open will succeed through the NAT (with

Agreed.

> If you have any specific concerns of the problems that would arise as a
> result of this, I'd like to know. (not to raise a discussion, but rather
> because I'm wondering what you are thinking of)

"big problems" refers to "if we needed ICMP to be timely" - since
routers tend to do ICMP processing somewhat off-line. I.e., I don't want
us to consider ICMPs that need to be timely, since that's never been a
requirement.

That issue also comes up if we talk about checking in-window of TCP
segments inside ICMPs (your ICMP-attacks I-D? I don't recall for sure
which doc, or whether it's currently still there), since there's no
requirement that the ICMP be issued within one RTT, so the window
therein has no meaning. I raised this issue several times before, most
recently in San Diego if I recall correctly (except for dinners, IETFs
tend to run together).

Joe

---

From: Saikat Guha <[email-masked]>
Date: Wed, 02 May 2007 17:31:52 -0400

> implemented, the simultaneous open will succeed through the NAT (with 

This is true assuming that 6 seconds is enough for the ends to
coordinate the setup. 6 seconds was really a number out of a hat; enough
for one SYN rexmit, and enough for vendors with embedded devices with
CPUs that took coffee breaks to feel comfortable with it.

In a rather contrived scenario where 6 seconds is not enough for
coordinating the SYNs, a stack (such as Windows) that ignores the ICMP
errors (hard errors even!) would succeed in traversing the NAT. In
another scenario, yet-to-be-published BEHAVE recommendations
notwithstanding, there exist deployed NATs today that instantly return
ICMP host unreachables (soft error).

I don't believe the approach proposed in your document needs to account
for cases where BEHAVE assumptions are violated (6 seconds not enough /
non BEHAVE compliant NAT) or the stack doesn't respect hard errors
(Windows). However, IMHO this interaction between NAT ICMPs, stacks that
process them, and its effect on NAT traversal is a potential pitfall /
source of problems that should be documented.

cheers,

---

From: Saikat Guha <[email-masked]>
Date: Wed, 02 May 2007 17:31:52 -0400

cheers,

---

From: Joe Touch <[email-masked]>
Date: Wed, 02 May 2007 16:11:53 -0700

> ICMP host unreachables (soft error).

Nice. Presumably they respond that way to all packets to their public
address? Otherwise, they're basically flaunting that they're flakey.

> source of problems that should be documented.

The behavior of a non-BEHAVE NAT or improper hard error processing
should be obvious, but we don't typically document such behavior except
when it's a pervasive issue, and we have called them "implementation
problems" (e.g., RFC 2525).

The fact that NATs might return host unreachable might be worth noting,
perhaps somewhere such as the "public side is a host; private side is a
router" doc I'm currently thinking of drafting (i.e., this is a place
where NATs violate that rule-of-thumb, and there are pitfalls). This
gives me further reason to work on that doc... ;-)

Joe

---

From: Fernando Gont <[email-masked]>
Date: Wed, 02 May 2007 20:21:29 -0300

At 06:31 p.m. 02/05/2007, Saikat Guha wrote:

>errors (hard errors even!) would succeed in traversing the NAT.

Windows honors hard errors for established connections. Do they 
ignore them for non-synchronized connections???

>In

Why do they send host unreachables instead of port unreachables?

>source of problems that should be documented.

Correct me if I am wrong:

The current BEHAVE requirements for TCP are that if an incoming SYN 
is received, and after 6 seconds a SYN is not seen the other way, an 
ICMP port unreachable is sent. This message is a hard error, to be 
processed during the conenction-establishment phase. Therefore, it 
does not really affect neither the soft errors draft nor the icmp 
attacks draft.

But there are some (non compliant) NATs out there that send icmp 
host(?) unreachables for the same scenario. In this case, if after 6 
seconds of receiving an incoming SYN an outgoing SYN is not seen by 
the NAT, a host unreachable would be sent, and thus the connection 
would be aborted (?). This is the scenario that should be mentioned?

Aside: Do implementations correctly handle simultaneous opens? Some 
time ago I read a paper with a proposal for dealing with SYN floods 
that I had played with a while ago: When you receive an incoming SYN, 
you send a SYN cookie *but* you don't set the ACK bit, and you don't 
keep state for this connection. The client will think that a 
simultaneous open took place, and thus retransmit its SYN (i.e., 
including all the same options as the original SYN). This way, you 
have SYN cookies, but you don't "lose" options. However, the author 
noted that many stacks did not handle "simultaneous opens" correctly. 
If anybody is interested, I can provide a pointer to the paper (would 
have to look on my HD)

Thanks,

---

From: Fernando Gont <[email-masked]>
Date: Wed, 02 May 2007 20:36:52 -0300

At 06:04 p.m. 02/05/2007, Joe Touch wrote:

(subject changed to reflect current discussion)

>us to consider ICMPs that need to be timely, since that's never been a

I understand. I was meaning if you could think of something that 
would "break" by adding this requirement.

>which doc, or whether it's currently still there),

Yes, icmp-attacks. (btw, I have asked the secretariat to repost the 
latest version, as the previous one expired)

>since there's no

Yes. I understand that there's currently no requirement for 
timeliness in ICMP. I'm just thinking: if there were, would anything break?

(In practice, it doesn't seem to break anything... as the sequence 
number check is implemented by virtually all implementations. 
However, I'm still interested in "what could theoretically break")

Thanks,

---

From: Joe Touch <[email-masked]>
Date: Wed, 02 May 2007 16:40:45 -0700

...
> Yes. I understand that there's currently no requirement for timeliness
> in ICMP. I'm just thinking: if there were, would anything break?

Except for existing implementations of routers and hosts that offload
ICMP to outboard processors?

I.e., it changes router design to have such a requirement.

> number check is implemented by virtually all implementations. However,
> I'm still interested in "what could theoretically break")

In practice, we don't stress the window much either. I.e., the reason
things don't break isn't because they're not on the verge of doing so.

Joe

---

From: Saikat Guha <[email-masked]>
Date: Wed, 02 May 2007 20:13:55 -0400

> ignore them for non-synchronized connections???

IIRC Windows XP (at least around March'05, pre-SP2) ignored ICMP errors
for connections in SYN-SENT state (certainly soft errors, but also hard
errors IIRC). That behavior may since have changed in SP2, which also
added/fixed support for TCP Simultaneous-Open.

> Why do they send host unreachables instead of port unreachables?

I wish I knew; NATs in the wild do all sorts of crazy things. 

> does not really affect neither the soft errors draft nor the icmp 

Correct. BEHAVE compliant NATs should not affect either of these
documents one way or another. 

> would be aborted (?). This is the scenario that should be mentioned?

Non compliant NATs are not bound by the 6 second rule. Some existing non
compliant NATs instantly send a soft error. As Joe mentioned, they may
simply be classified as an implementation problem; or may be called out
in the soft errors document if it is in scope.

> Aside: Do implementations correctly handle simultaneous opens? 

Windows XP pre-SP2 -- No
Windows XP SP2, Linux, OpenBSD -- Yes
Vista, MaxOS, others -- Not tested

Aside from the host OS, many NATs (32%) do not support TCP S-O, although
some firmwares have since been updated to add support for it.
(https://www.guha.cc/saikat/stunt-results.php?sort=24 Incoming SYN after
outgoing SYN column)

---

From: Pasi Sarolahti <[email-masked]>
Date: Thu, 03 May 2007 15:24:05 +0300

On Tue, 2007-04-24 at 23:46 -0400, ext Mark Allman wrote:
> > Mark and I would like to start a WG last call on the informational
> > document on TCP's reaction to soft ICMP errors on connection set-up:

I read the latest version of soft-errors-draft, and overall the draft
looks ok to me. If I remember it correctly, earlier there were some
concerns about the language being too strong for an informational
document, but the current version seems ok also in this respect. The
draft is in fact quite explicit that the documented behavior is in
conflict with the standard TCP, and no change to the current standard is
proposed. I also think this is a useful draft, because it describes a
relevant problem in address resolution for dual-stack/multihomed hosts,
and documents a widely implemented workaround.

Some additional comments below. Many of them are
stylish issues that can be ignored if you disagree about them.

Abstract:

>    This document describes a non-standard, but widely implemented,
>    modification to TCP's handling of ICMP soft error messages received
>    in any of the non-synchronized states, that rejects connections
>    experiencing those errors immediately.  This behavior reduces the

  The first sentence of abstract is a bit unclear. How about: "...that
  immediately rejects connections experiencing ICMP soft errors."

Section 1., paragraph 4:

>    This document analyzes the fault recovery strategy of TCP [RFC0793],
>    and the problems that may arise due to TCP's reaction to ICMP soft
>    errors.  Among others, it analyzes the problems that may arise in
>    scenarios where dual stack nodes that have IPv6 enabled by default
>    are deployed in IPv4 or mixed IPv4 and IPv6 environments.

  It could be useful to mention here that the same problem and the
  same workaround also applies in general for multihomed
  destinations that can be reached by multiple addresses.

Section 2, paragraph 2:

>    The Host Requirements RFC [RFC1122] states, in Section 4.2.3.9., that
>    the ICMP messages that indicate soft errors are ICMP "Destination
>    Unreachable" codes 0 (network unreachable), 1 (host unreachable), and
>    5 (source route failed), ICMP "Time Exceeded" codes 0 (time to live
>    exceeded in transit) and 1 (fragment reassembly time exceeded), and
>    ICMP "Parameter Problem".  Even though ICMPv6 didn't exist when
>    [RFC1122] was written, one could extrapolate the concept of soft
>    errors to ICMPv6 "Destination Unreachable" codes 0 (no route to
>    destination) and 3 (address unreachable), ICMPv6 "Time Exceeded"
>    codes 0 (Hop limit exceeded in transit) and 1 (Fragment reassembly
>    time exceeded), and ICMPv6 "Parameter Problem" codes 0 (Erroneous
>    header field encountered), 1 (Unrecognized Next Header type
>    encountered) and 2 (Unrecognized IPv6 option encountered).

  Should have reference to ICMPv6.

Section 2, paragraph 5:

>    TCP's reaction to ICMP messages will depend on the type of error
>    being signaled.

  This and the previous two paragraphs each consist of only one
  sentence each (and in fact, many other paragraphs in the draft). For
  example, I wonder if the two paragraphs before this one could
  be combined.

Section 4., paragraph 1:

>    As discussed in Section 1, it may make sense for the fault recovery

  I would just leave out "As discussed in Section 1", since IMHO this
  is not so much discussed in the intro.

Section 4., paragraph 5:

>    We note that the TCPM WG could not arrive at consensus on allowing
>    the above described behavior as part of the standard.  Therefore,
>    treating soft errors as hard errors during connection establishment,
>    while widespread, is not part of standard TCP behavior and this
>    document does not change that state of affairs.

  I would reorganize Sections 4 and 5 to be subsections of one
  top-level section. The top-level section could be titled similarly
  than section 4 is now, except using "workarounds" instead of singular
  form. Section 4.1 could be something like "Simple workaround",
  and 4.2 would still keep "A More Conservative Approach". I would
  move the above note to the beginning of the top-level section, to
  immediately notify the reader that the draft just documents the
  known implementations, and does not propose these to be part of
  standard TCP. This also deserves a bit more explanation on why TCPM
  is not proposing this to be part of standard TCP. Maybe a forward
  reference to "Possible drawbacks" section would help here.

Section 5., paragraph 5:

>    This approach would give the network more time to solve the
>    connectivity problem than simply aborting a connection attempt upon
>    reception of the first soft error.  However, it should be noted that
>    depending on the values chosen for the MAXSYNREXMIT and MAXSOFTERROR
>    parameters, this approach could still lead to long delays between
>    connection establishment attempts, thus not solving the problem.  For
>    example, BSD systems abort connections in the SYN-SENT or the SYN-
>    RECEIVED state when a second ICMP error is received, and the SYN
>    segment has been retransmitted more than three times.  They also set
>    up a "connection-establishment timer" that imposes an upper limit on
>    the time the connection establishment attempt has to succeed, which
>    expires after 75 seconds [Stevens2] (pp. 828-829).  Even when this
>    policy may be better than the three-minutes timeout policy specified
>    in [RFC1122], it may still be inappropriate for handling the
>    potential problems described in this document.  This more
>    conservative approach has been implemented in BSD systems since, at
>    least, 1994 [Stevens2].

  "since, at least, 1994." seems a little odd. Maybe just "for more
  than 10 years"?

Section 6., paragraph 1:

>    The following subsections discuss some of the possible drawbacks
>    arising from the use of the non-standard modifications to TCP's
>    reaction to soft errors described in Section 4 and Section 5.

  The drawbacks section seems quite short compared to the amount
  of discussion TCPM had earlier on this draft, although I don't
  have much to add here off the top of my head. If this section
  appropriately summarizes the arguments against making this change
  to TCP, then I guess it is ok.

  One minor point could be added to 6.1 that in many cases you only
  get one address for a destination, in which case it might have
  been better to try that address persistently according to normal
  TCP rules, instead of just aborting the connection establishment.

On normative references (section 10.1):
  This is a dumb question, not a comment... Is Informative RFC
  supposed to have Normative References? Or should all references
  be informative?

- Pasi

---

From: Ted Faber <[email-masked]>
Date: Thu, 3 May 2007 09:44:12 -0700

On Wed, May 02, 2007 at 08:45:23AM -0400, Saikat Guha wrote:
> queue.

This seems to have generated some discussion, and seems to me (with
my chair hat) to be an interaction worth mentioning in the tcpm draft.
I think that would be a lot easier to accomodate (i.e., easy to add
without another WGLC round) if someone were to suggest a paragraph or 2
to be included in the tcpm draft and we hammered out a quick consensus.

Any takers?

(And if I'm reading the group's take wrong here, let me know that, too.)

-- 
Ted Faber
http://www.isi.edu/~faber           PGP: http://www.isi.edu/~faber/pubkeys.asc
Unexpected attachment on this mail? See http://www.isi.edu/~faber/FAQ.html#SIG
_______________________________________________
tcpm mailing list
[email-masked]
https://www1.ietf.org/mailman/listinfo/tcpm

---

From: Fernando Gont <[email-masked]>
Date: Fri, 04 May 2007 05:12:35 -0300

>(And if I'm reading the group's take wrong here, let me know that, too.)

So far Saikat is for documenting this, and Joe for not doing so.

If the decision is to document this, then a possible text would be:

Some NATs respond to an unsolicited inbound SYN packet with an ICMP 
soft error message. If the system sending the unsolicited SYN segment 
implements the workaround described in this document, it will abort 
the connection upon receipt of the ICMP soft error, thus preventing 
the connection establishment from succeeding. It should be stressed 
that those NATs implementing this behaviour are not BEHAVE-compliant, 
and should implement REQ-4 of [behave-tcp] instead.

That's my two cents. Improvements to the text may probably be necessary.

Kindest regards,

---

From: Joe Touch <[email-masked]>
Date: Fri, 04 May 2007 15:34:01 -0700

> So far Saikat is for documenting this, and Joe for not doing so.

Largely because it focuses on behavior that is not BEHAVE compliant;
IMO, this sort of thing should be for a BEHAVE-based 'implementation
problems' doc. If we want to add this, are we adding all other known
ICMP implementation errors?

> Some NATs respond to an unsolicited inbound SYN packet with an ICMP soft
> error message. If the system sending the unsolicited SYN segment
> implements the workaround described in this document, it will abort the
> connection upon receipt of the ICMP soft error, thus preventing the
> connection establishment from succeeding. It should be stressed that
> those NATs implementing this behaviour are not BEHAVE-compliant, and
> should implement REQ-4 of [behave-tcp] instead.
>

---

From: Ted Faber <[email-masked]>
Date: Tue, 8 May 2007 11:57:28 -0700

> ICMP implementation errors?

It seems to me (hat's off) that a pitfall that this soft-errors response
causes when combined with an IETF BCP is a more constrained and
important case than an interaction between this response and a buggy
system.  I agree it needs to be flagged in the NAT literature as a
pitfall for BEHAVE NAT implementers.  I think it should also be flagged
here for TCP implementers who want to understand the ramifications of
this design choice (soft error == hard error during connection setup).
It's worth calling out against all the other potential design choices
because it conflicts with a BCP - implementation advice - rather than
just against observed behavior.

In the same way the soft-errs doc points out that a TCP that treats soft
errors as hard is out of step with ICMP and TCP standards, I think it
should point out that the decision violates the assumptions of the
BEHAVE BCP.

Anyone else care?  If it's just Saikat and me who think this warning
needs to be added, I'm not opposed to having the draft go to RFC as-is.

---

From: Ted Faber <[email-masked]>
Date: Tue, 8 May 2007 14:12:04 -0700

On Tue, Apr 24, 2007 at 11:46:24PM -0400, Mark Allman wrote:
> > 	Pages           : 13
> > 	Date            : 2007-4-4
> > http://www.ietf.org/internet-drafts/draft-ietf-tcpm-tcp-soft-errors-05.txt
> > 
> > Our understanding is that all feedback to date is reflected in this
> > version and that there are no known outstanding issues.  Please send
> > feedback (even if just of the form "yep, looks good").
> > 
> > This WGLC will end on Monday 23 Apr 2007.
> 
> Folks-
> 
> We received nearly no feedback on this WGLC.  Since this draft has been
> contentious in the past we need some review before passing it along to
> the IESG.  We are therefore extending the WGLC until May 15.  Please
> take a few minutes and look over the draft and send comments.  As Ted
> notes above, even if you think the draft looks fine a one-sentence note
> that says this would be very much appreciated.

May 15 is a week away.

If you have comments, please send them.

---

From: Saikat Guha <[email-masked]>
Date: Wed, 09 May 2007 17:39:21 -0400

> needs to be added, I'm not opposed to having the draft go to RFC

Ditto.