Subject: [Simple] Objects to NATs kill TCP connection

From: Cullen Jennings <[email-masked]>
Date: Sat, 15 Nov 2003 16:31:56 -0800

There was some assertion in the WG meeting that NATs arbitrarily kill TCP
connection and the connection needs to be set up again. I'm sure this has
happened with broken NATS but I do not feel it is widespread and don't think
we should specifically design for it. I do think we should design for
graceful failure or recover when things in the network fail but I don't
think there is a common case of TCP failing that involves NATs. If there
were, many things that do not automatically set up a new TCP connection,
like ssh for example, would be nearly unusable through these broken NATs.

Cullen

_______________________________________________
Simple mailing list
[email-masked]
https://www1.ietf.org/mailman/listinfo/simple

---

From: Paul Kyzivat <[email-masked]>
Date: Mon, 17 Nov 2003 09:22:10 -0500

Cullen,

I don't have any data of my own on this subject, but I hope you are right.

It would be a lot simpler if we can continue to assume that the way you 
reestablish an MSRP session is via a reINVITE. The adequacy of that was 
under question if NATs regularly blow away connections.

	Paul

---

From: Paul Kyzivat <[email-masked]>
Date: Mon, 17 Nov 2003 09:22:10 -0500

Cullen,

	Paul

---

From: Ben Campbell <[email-masked]>
Date: Mon, 17 Nov 2003 13:37:37 -0600

I have personally encountered this behavior in the wild. I agree, 
however, that it was broken, and probably not a normal use case. And as 
far as I know, it pretty much killed anything other than hit-and-run 
HTTP connections.

---

From: Ben Campbell <[email-masked]>
Date: Mon, 17 Nov 2003 13:37:37 -0600

---

From: Jonathan Rosenberg <[email-masked]>
Date: Tue, 18 Nov 2003 12:13:07 -0500

Short timeouts are broken, sure, but the fact that NATs have timeouts 
on bindings is a common behavior. As a result, if there is no traffic 
on the TCP connection before the timeout occurs, the binding is lost, 
and any data sent on that connection will fail.

Therefore, I think we can expect to see cases where the connections 
close due to NAT.

Now, the question is, do we think we would see periods of inactivity 
on the messaging will exceed a typical nat timeout? I think they may. 
I personally tend to leave IM sessions open continuously all day..

-Jonathan R.

Ben Campbell wrote:

>> It would be a lot simpler if we can continue to assume that the way 
>> you reestablish an MSRP session is via a reINVITE. The adequacy of 
>> that was under question if NATs regularly blow away connections.
>>
>>     Paul
>>
>> Cullen Jennings wrote:
>>

-- 
Jonathan D. Rosenberg, Ph.D.                600 Lanidex Plaza
Chief Technology Officer                    Parsippany, NJ 07054-2711
dynamicsoft
[email-masked]                     FAX:   (973) 952-5050
http://www.jdrosen.net                      PHONE: (973) 952-5000
http://www.dynamicsoft.com

---

From: Jonathan Rosenberg <[email-masked]>
Date: Tue, 18 Nov 2003 12:13:07 -0500

-Jonathan R.

---

From: Ben Campbell <[email-masked]>
Date: Tue, 18 Nov 2003 12:11:53 -0600

When we were having the recent discussion on the inactivity timeout for 
MS(R)P, I don't think we thought much about the fact that MSRP keepalive 
messages may also act as NAT binding keepalives.

Jonathan Rosenberg wrote:

> Short timeouts are broken, sure, but the fact that NATs have timeouts on 
> bindings is a common behavior. As a result, if there is no traffic on 
> the TCP connection before the timeout occurs, the binding is lost, and 
> any data sent on that connection will fail.
> 
> Now, the question is, do we think we would see periods of inactivity on 
> the messaging will exceed a typical nat timeout? I think they may. I 
> personally tend to leave IM sessions open continuously all day..
> 
> -Jonathan R.
> 
> Ben Campbell wrote:
> 
>> however, that it was broken, and probably not a normal use case. And 
>> as far as I know, it pretty much killed anything other than 
>> hit-and-run HTTP connections.
>>
>> Paul Kyzivat wrote:
>>
>>> Cullen,
>>>

---

From: Ben Campbell <[email-masked]>
Date: Tue, 18 Nov 2003 12:11:53 -0600

---

From: Jonathan Rosenberg <[email-masked]>
Date: Tue, 18 Nov 2003 13:17:00 -0500

The trick, as always, is knowing what the nat keepalive is. There is 
no easy way to know.

-Jonathan R.

---

From: Jonathan Rosenberg <[email-masked]>
Date: Tue, 18 Nov 2003 13:17:00 -0500

-Jonathan R.

---

From: Paul Kyzivat <[email-masked]>
Date: Tue, 18 Nov 2003 13:30:27 -0500

Letting msrp keepalives act as NAT binding keepalives seems attractive. 
Perhaps the NAT keepalive time needs to be determined adaptively. If the 
connection is found to be down, a reinvite could reconstruct it, and a 
new keepalive timer could be set, shorter than before.

But this would require negotiating the keepalive time.

	Paul

Jonathan Rosenberg wrote:
> The trick, as always, is knowing what the nat keepalive is. There is no 
> easy way to know.
> 
> -Jonathan R.
> 
> Ben Campbell wrote:
> 
>> When we were having the recent discussion on the inactivity timeout 
>> for MS(R)P, I don't think we thought much about the fact that MSRP 
>> keepalive messages may also act as NAT binding keepalives.
>>
>>
>>

---

From: Paul Kyzivat <[email-masked]>
Date: Tue, 18 Nov 2003 13:30:27 -0500

Paul

---

From: Ben Campbell <[email-masked]>
Date: Tue, 18 Nov 2003 12:35:04 -0600

Can we reasonably assume that all connection failures are due to NATs? 
What happens if endpoints do this when the failure is caused by 
something else?

Paul Kyzivat wrote:

>>>> Short timeouts are broken, sure, but the fact that NATs have 
>>>> timeouts on bindings is a common behavior. As a result, if there is 
>>>> no traffic on the TCP connection before the timeout occurs, the 
>>>> binding is lost, and any data sent on that connection will fail.
>>>>
>>>> on the messaging will exceed a typical nat timeout? I think they 
>>>> may. I personally tend to leave IM sessions open continuously all day..
>>>>
>>>> -Jonathan R.
>>>>
>>>> Ben Campbell wrote:
>>>>
>>>>> however, that it was broken, and probably not a normal use case. 
>>>>> And as far as I know, it pretty much killed anything other than 
>>>>>> It would be a lot simpler if we can continue to assume that the 
>>>>>> way you reestablish an MSRP session is via a reINVITE. The 
>>>>>> adequacy of that was under question if NATs regularly blow away 
>>>>>> connections.
>>>>>>
>>>>>>     Paul
>>>>>>
>>>>>> Cullen Jennings wrote:
>>>>>>

---

From: Ben Campbell <[email-masked]>
Date: Tue, 18 Nov 2003 12:35:04 -0600

---

From: Paul Kyzivat <[email-masked]>
Date: Tue, 18 Nov 2003 14:12:41 -0500

> What happens if endpoints do this when the failure is caused by 

Good point. That could be ugly - I don't see any easy way to know when 
the connection fails because of NAT. Lacking that it seems infeasible to 
test for the NAT timeout interval.

Are there any standard practices in this regard? In other words, is 
there a keepalive time we could use that would be sufficient most of the 
time and still not be too impractical to keep up?

	Paul

> Paul Kyzivat wrote:
> 
>> Letting msrp keepalives act as NAT binding keepalives seems 
>> attractive. Perhaps the NAT keepalive time needs to be determined 
>> adaptively. If the connection is found to be down, a reinvite could 
>> reconstruct it, and a new keepalive timer could be set, shorter than 
>> before.
>>
>>>>> Now, the question is, do we think we would see periods of 
>>>>> inactivity on the messaging will exceed a typical nat timeout? I 
>>>>> think they may. I personally tend to leave IM sessions open 
>>>>> continuously all day..
>>>>>
>>>>> -Jonathan R.
>>>>>
>>>>> Ben Campbell wrote:
>>>>>

---

From: Paul Kyzivat <[email-masked]>
Date: Tue, 18 Nov 2003 14:12:41 -0500

Paul

---

From: "Dean Willis" <[email-masked]>
Date: Fri, 21 Nov 2003 16:30:32 -0600

> 
> There was some assertion in the WG meeting that NATs 
> arbitrarily kill TCP connection and the connection needs to 
> be set up again. I'm sure this has happened with broken NATS 
> but I do not feel it is widespread and don't think we should 
> specifically design for it. I do think we should design for 
> graceful failure or recover when things in the network fail 
> but I don't think there is a common case of TCP failing that 
> involves NATs. If there were, many things that do not 
> automatically set up a new TCP connection, like ssh for 
> example, would be nearly unusable through these broken NATs.

Usually when I'm sittig in a 3GPP2 meeting trying to do email and what not
over an SSH tunnel I run a script on the server-side of the ssh tunnel that
prints a "."  every ten seconds.

Why do I do this? 

Because if I don't, the freaking NAT box (usually a Cisco/Linksys product,
set to defaults I believe) drops my port binding between mail polls, and I
have to restart the ssh connection. The TCP keepalives aren't enough
frequent enough to keep the NAT alive. And restarting ssh si a manual
process, requiring re-entry of a key's secret.

Translation: ssh IS nearly unusable with these broken NATs, unless one is
keeping it almost constantly busy. MSRP sessions would have very similar
properties.

I believe this is demonstrably provable whenever the NAT reclamation timer
is shorter than the longest-case interpacket timer, which is generally TCP
keepalives at about ten minutes . . . And considering that most
off-the-shelf NAT/routers  seem to run reclamation timers of between thirty
seconds and three minutes, it's a problem.

--
Dean

---

From: "Dean Willis" <[email-masked]>
Date: Fri, 21 Nov 2003 16:30:32 -0600

Why do I do this? 

--
Dean

---

From: Cullen Jennings <[email-masked]>
Date: Mon, 24 Nov 2003 09:42:45 -0800

I went and did some testing with a handful of different NATs and poked
around on documentation for more. Here's the initial results I found - if
folks can point me to stuff that contradicts this, I would be grateful ....

Just about every OS I can muck with, (Windows 98,2000,XP,  Linux, Solaris,
HPUX) seems to set the default TCP keep alive timer to 2 hours (7200
seconds). Due to the multiples retires and such it takes several minutes
after this before anything is really killed.

The NAT and FW I looked at (mostly Cisco and Linksys but few others too)
seem to also set the TCP timeouts at 2 hours plus a bit.

This is probably not a coincidence. It is a little disturbing that they are
close to the same instead NAT/FW being significantly above the OS default
but I'm still trying to understand that.

The two hours makes in not much fun to run tests on this stuff :-)

Cullen

PS - my example of SSH was lousy because many SSH clients allow you to set
up application level keep alives.

---

From: Cullen Jennings <[email-masked]>
Date: Mon, 24 Nov 2003 09:42:45 -0800

Cullen

---

From: Jonathan Rosenberg <[email-masked]>
Date: Mon, 24 Nov 2003 16:36:54 -0500

> seem to also set the TCP timeouts at 2 hours plus a bit.

But these are configurable, right? The issue, unfortunately, is what 
is out there in the real world. We had a nasty experience with a large 
enterprise that had them set really low, and I am wondering how 
frequently these timeouts are adjusted down in the interests of 
"security".

-Jonathan R.

---

From: Jonathan Rosenberg <[email-masked]>
Date: Mon, 24 Nov 2003 16:36:54 -0500

-Jonathan R.

---

From: Cullen Jennings <[email-masked]>
Date: Mon, 24 Nov 2003 16:52:33 -0800

> frequently these timeouts are adjusted down in the interests of

Good question - I don't know. Some (cough cough) organizations have
configured their firewall to not allow any UDP through which puts a real
damper on VoIP.

As the saying goes, I see 10 types of people

a) The ones that have not changed the settings from the defaults

b) The ones that have changed the settings. Hopefully these are capable of
changing them to setting that will work if they really feel like it. If they
don't feel like it, them I have no interest in trying to circumvent the
policy they want to apply.

I suspect that really short lived TCP timeouts are not extremely common but
have no way to prove that one way or another.

Cullen

---

From: Cullen Jennings <[email-masked]>
Date: Mon, 24 Nov 2003 16:52:33 -0800

Cullen

---

From: Henning Schulzrinne <[email-masked]>
Date: Mon, 24 Nov 2003 21:42:57 -0500

I've just experienced a NAT/SIP-enabled DSL firewall that had a default 
value of 5 minutes of inactivity timeout, fortunately configurable.

---

From: Henning Schulzrinne <[email-masked]>
Date: Mon, 24 Nov 2003 21:42:57 -0500