Subject: Re: [tcpm] host behaviour when connectivity is lost

From: Mikael Abrahamsson <[email-masked]>
Date: Tue, 1 Jan 2013 18:29:00 +0100 (CET)

On Fri, 9 Nov 2012, Mikael Abrahamsson wrote:

I have sent this email both to the list (as can be seen below), and to one 
of the chairs 5 days later, with zero reply. I would really appricate some 
kind of feedback on this idea, if nothing else just a "this is not 
applicable to this WG" and perhaps a hint on where it would be more 
applicable.

Thanks.

>
> Hello.
>
> I don't know if this is appropriate for this WG, please tell me if it's not.
>
> A while ago, I pitched an idea in the linux-netdev mailing list, regarding 
> how a connection manager might hint the TCP stack of the host, to do things 
> depending on network conditions.
>
> The post is here: <http://www.spinics.net/lists/netdev/msg213615.html>
>
> Would what I am suggesting be appropriate to discuss here and perhaps even 
> create a document giving guidance to TCP implementors about APIs that might 
> be helpful, and how they should behave? I believe the problem I'm describing 
> is real and it affects users all the time.
>
> Here is the text:
>
> ------------------------------------
>
> I'm a network engineer, as in I work primarily with IP routing. Ever since I 
> started running Linux back in the mid 90ties I've had a love/hate 
> relationship with how Linux handles disappearing network connectivity or IP 
> addresses.
>
>
> In my mind there are two ways to handle outage:
>
> 1. When a network connection (physical interface) goes down, keep everything 
> as it is, it might come back up again shortly and then we can continue as if 
> basically nothing happened. TCP was designed for this considering timeouts 
> can be in hours.
>
>
> 2. When a network connection (physical interface) goes down, wait a few 
> seconds, give up, reset all connectivity related to that connection, 
> basically give up.
>
>
> Now to my question for the netdev people:
>
> Is there functionality in the kernel for a connection manager to easily 
> accomplish 2, in that when it tries to deconfigure the IP address on the 
> interface to also kill all TCP connections terminated at that IP? On my 
> laptop, I regularily have to kill my ssh client after suspend/resume cycle, 
> because it's been down for quite a while, and the ssh client doesn't know the 
> TCP connection is now not functional anymore (TCP session is still up and 
> retransmit won't happen for a while, so the TCP RST from the server (I use 
> keepalives within SSH) isn't seen for a long time).
>
>
> Without knowing what's in place right now, I see some behaviours that I'd 
> like to have:
>
>
> After resume (or otherwise network connectivity re-established), connection 
> manager should be able to tell the kernel to:
>
>
> a) kill all TCP/UDP/other sessions existing which doesn't currently have an 
> active IP address on the machine. This is for the sake of local clients. b) 
> the TCP/SCTP sessions that *do* have an IP, should have their retransmit 
> timers "reset", so that whatever needs to be sent, should be sent immediately 
> (or shortly, within a few seconds). c) tell the kernel to kill all TCP 
> sessions bound to a certain IP, because the connection manager is going to 
> remove it shortly. Send TCP RSTs or whatever and close the TCP session, so 
> both ends know that network connectivity is going down.
>
>
> This would mean that if I resume my laptop and it establishes network 
> connectivity, I will then *know* within a few seconds what TCP sessions are 
> still valid (the ones that aren't will be killed either because they're bound 
> to an IP that is not available, or a TCP packet is sent out to which there 
> will be a TCP RST reply from the other side if the TCP session is closed at 
> that end).
>
>
> All of this also has implications on IPv6 and autoconfiguration, I don't know 
> if currently we are closing TCP sessions bound to IPv6 addresses that are 
> going away due to the RA the addresses were autoconfigured based on, now 
> doesn't have a valid lifetime anymore and the kernel is going to remove them. 
> It also applies to both DHCPv6 and DHCPv4 when a lease is expiring and the 
> IPv4/IPv6 address is going to be removed.
>
>
> Right now I think the situation with a lot of "hanging" TCP sessions after 
> connectivity is restored is suboptimal and negatively impacts user 
> experience. Probably there should be knobs to turn for different user needs 
> (server or desktop/laptop) because desired behaviour is different in 
> different use cases.
>
> -- 
> Mikael Abrahamsson    email: [email-masked]
> _______________________________________________
> tcpm mailing list
> [email-masked]
> https://www.ietf.org/mailman/listinfo/tcpm
>

-- 
Mikael Abrahamsson    email: [email-masked]

---

From: "Scharf, Michael (Michael)" <[email-masked]>
Date: Tue, 1 Jan 2013 19:52:30 +0100

Mikael,

The lack of a reply is feedback as well...

My understanding (as individual WG contributor, for what it is worth) is that one has to distinguish functions inside the TCP stack and user/system programs using TCP. From a high-level perspective, TCPM cares about the former, but not about the latter.

A connection manager is in my understanding a function *on top* of the TCP/IP network stack. Connection managers are realized already today in operating systems, i. e., the key question is whether any additional TCP (or IETF) standard is actually required. Note that the IETF typically does not standardize operating system interfaces. 

More specifically regarding your suggestions:

Proposals a) and c) are about killing TCP connections. In my understanding, a TCP implementation itself cannot decide whether there is "an active IP address on the machine", i. e., it is apparently not up to the TCP implementation to decide this. Moreover, as you already note yourself, the question whether it is "better" to persist or kill a connection is highly application-specific, i. e., TCP cannot decide that at all. To me, this is an application decision that TCP cannot solve. For instance, many application protocols have means to check whether a connection is still alive, and they also have their own means to react to connectivity problems (I don't know about SSH). That seems to be the right solution, and does not require TCP modifications, imho.

A potentially interesting field that could be relevant to TCPM is whether TCP's timer management can be further improved after a system resume, i. e., bullet point b) in your list. However, my assumption is that the typical implementations of retransmission timers inside an OS automatically fire after a system resume. If so, objective b) might already be realized by TCP stacks as of today. But I might be wrong.

In short, in TCPM you should explain what is indeed missing in the TCP standards.

Michael

> -----Original Message-----
> From: [email-masked] [mailto:[email-masked]] On 
> Behalf Of Mikael Abrahamsson
> Sent: Tuesday, January 01, 2013 6:29 PM
> To: [email-masked]
> Subject: Re: [tcpm] host behaviour when connectivity is lost
> 
> I have sent this email both to the list (as can be seen 
> below), and to one of the chairs 5 days later, with zero 
> reply. I would really appricate some kind of feedback on this 
> idea, if nothing else just a "this is not applicable to this 
> WG" and perhaps a hint on where it would be more applicable.
> 
> Thanks.
> 
> >
> > Hello.
> >
> > A while ago, I pitched an idea in the linux-netdev mailing list, 
> > regarding how a connection manager might hint the TCP stack of the 
> > host, to do things depending on network conditions.
> >
> > The post is here: 
> <http://www.spinics.net/lists/netdev/msg213615.html>
> >
> > Would what I am suggesting be appropriate to discuss here 
> and perhaps 
> > even create a document giving guidance to TCP implementors 
> about APIs 
> > that might be helpful, and how they should behave? I believe the 
> > problem I'm describing is real and it affects users all the time.
> >
> > Here is the text:
> >
> > I'm a network engineer, as in I work primarily with IP 
> routing. Ever 
> > since I started running Linux back in the mid 90ties I've had a 
> > love/hate relationship with how Linux handles disappearing network 
> > connectivity or IP addresses.
> >
> >
> > 1. When a network connection (physical interface) goes down, keep 
> > everything as it is, it might come back up again shortly 
> and then we 
> > can continue as if basically nothing happened. TCP was designed for 
> > this considering timeouts can be in hours.
> >
> >
> > 2. When a network connection (physical interface) goes down, wait a 
> > few seconds, give up, reset all connectivity related to that 
> > connection, basically give up.
> >
> >
> > Is there functionality in the kernel for a connection manager to 
> > easily accomplish 2, in that when it tries to deconfigure the IP 
> > address on the interface to also kill all TCP connections 
> terminated 
> > at that IP? On my laptop, I regularily have to kill my ssh client 
> > after suspend/resume cycle, because it's been down for 
> quite a while, 
> > and the ssh client doesn't know the TCP connection is now not 
> > functional anymore (TCP session is still up and retransmit won't 
> > happen for a while, so the TCP RST from the server (I use 
> > Without knowing what's in place right now, I see some 
> behaviours that 
> > I'd like to have:
> >
> >
> > After resume (or otherwise network connectivity re-established), 
> > connection manager should be able to tell the kernel to:
> >
> >
> > a) kill all TCP/UDP/other sessions existing which doesn't currently 
> > have an active IP address on the machine. This is for the sake of 
> > local clients. b) the TCP/SCTP sessions that *do* have an 
> IP, should 
> > have their retransmit timers "reset", so that whatever needs to be 
> > sent, should be sent immediately (or shortly, within a few 
> seconds). 
> > c) tell the kernel to kill all TCP sessions bound to a certain IP, 
> > because the connection manager is going to remove it 
> shortly. Send TCP 
> > RSTs or whatever and close the TCP session, so both ends 
> know that network connectivity is going down.
> >
> >
> > connectivity, I will then *know* within a few seconds what TCP 
> > sessions are still valid (the ones that aren't will be 
> killed either 
> > because they're bound to an IP that is not available, or a 
> TCP packet 
> > is sent out to which there will be a TCP RST reply from the 
> other side 
> > if the TCP session is closed at that end).
> >
> >
> > All of this also has implications on IPv6 and autoconfiguration, I 
> > don't know if currently we are closing TCP sessions bound to IPv6 
> > addresses that are going away due to the RA the addresses were 
> > autoconfigured based on, now doesn't have a valid lifetime 
> anymore and the kernel is going to remove them.
> > It also applies to both DHCPv6 and DHCPv4 when a lease is 
> expiring and 
> > the
> > Right now I think the situation with a lot of "hanging" TCP 
> sessions 
> > after connectivity is restored is suboptimal and negatively impacts 
> > user experience. Probably there should be knobs to turn for 
> different 
> > user needs (server or desktop/laptop) because desired behaviour is 
> > different in different use cases.
> >
> > --

---

From: Mikael Abrahamsson <[email-masked]>
Date: Wed, 2 Jan 2013 05:06:42 +0100 (CET)

On Tue, 1 Jan 2013, Scharf, Michael (Michael) wrote:

> A potentially interesting field that could be relevant to TCPM is 
> whether TCP's timer management can be further improved after a system 
> resume, i. e., bullet point b) in your list. However, my assumption is 
> that the typical implementations of retransmission timers inside an OS 
> automatically fire after a system resume. If so, objective b) might 
> already be realized by TCP stacks as of today. But I might be wrong.

My thinking is if there is TCP standardization/guidance needed for what 
the TCP timers should be set to when a system is resumed from hibernation 
or when know network loss is rectified.

There has been a lot of discussion on retransmission timers for SYN+ACK 
packets here recently, I feel that giving guidance on what timers should 
be after prolonged network connectivity loss or sleep as well. Keeping it 
at real time clock values what it should have been if this was just silent 
packet loss seems overly conservative (we know network was out, it's now 
most likely not, keeping retransmission timers in the tens of minutes is 
not helping anyone), and putting them into repeatedly fast-retransmit 
seems overly aggressive. Off the top of my head, I'd say putting them into 
somewhere around 1 second retransmit time range sounds plausable. But this 
is what I want to discuss.

> In short, in TCPM you should explain what is indeed missing in the TCP standards.

Guidance on timers, at least. Perhaps also guidance that having an 
operating system interface for userspace applications (perhaps a 
connection manager) is a good idea to indicate that TCP timers should be 
put into "network connectivity problem gone, please try to see what TCP 
sessions are still valid" (and how to do this) and have TCP lower timers 
and send packets that need sending, in a reasonable time (and define what 
is reasonable).

---

From: "Scharf, Michael (Michael)" <[email-masked]>
Date: Wed, 2 Jan 2013 10:02:34 +0100

> My thinking is if there is TCP standardization/guidance 
> needed for what the TCP timers should be set to when a system 
> is resumed from hibernation or when know network loss is rectified.

I think that these are two different cases.

There has been a lot of work on improving TCP during periods of disconnectivity in the last decade, mostly driven by mobile use. In contrast, I am not really aware of work on hibernation.

> There has been a lot of discussion on retransmission timers 
> for SYN+ACK packets here recently, I feel that giving 
> guidance on what timers should be after prolonged network 
> connectivity loss or sleep as well. Keeping it at real time 
> clock values what it should have been if this was just silent 
> packet loss seems overly conservative (we know network was 
> out, it's now most likely not, keeping retransmission timers 

Well, a TCP instance can hardly know this without cross-layer triggers - yet another area that was (not really sucessfully) investigated a lot in the last decade.

> in the tens of minutes is not helping anyone), and putting 
> them into repeatedly fast-retransmit seems overly aggressive. 
> Off the top of my head, I'd say putting them into somewhere 
> around 1 second retransmit time range sounds plausable. But 
> this is what I want to discuss.

Before discussing solutions, it would make sense to provide first real-world data showing that there is indeed a TCP problem.

For instance, to me it would be really useful to see some explanation how relavant TCP stacks deal with hibernation, and how suboptimal their timers are in reality. Also, potential guidance in that space has to be realistic, i. e., it must take into account the actual constraints inside a real TCP stack. So, we have to understand these constraints.

Michael

> Guidance on timers, at least. Perhaps also guidance that 
> having an operating system interface for userspace 
> applications (perhaps a connection manager) is a good idea to 
> indicate that TCP timers should be put into "network 
> connectivity problem gone, please try to see what TCP 
> sessions are still valid" (and how to do this) and have TCP 
> lower timers and send packets that need sending, in a 
> reasonable time (and define what is reasonable).
> 
> --

---

From: Mikael Abrahamsson <[email-masked]>
Date: Wed, 2 Jan 2013 14:06:32 +0100 (CET)

On Wed, 2 Jan 2013, Scharf, Michael (Michael) wrote:

> Before discussing solutions, it would make sense to provide first 
> real-world data showing that there is indeed a TCP problem.

Well, as I stated in my original email, I have this problem all the time 
when I resume my Ubuntu 12.04 machine (and the behaviour has been 
identical since 8.04 or 8.10 when I first started running ubuntu on this 
laptop), I have to kill my ssh client because TCP is "stuck". The other 
end has long ago closed its end of the session, but my end hasn't noticed 
and even if I wait several minutes, it won't.

My experience is that Windows (XP and 7) in case of resume will just kill 
all TCP sessions, which I don't think is optimal either.

I know several applications that run their own keep-alive on top of TCP 
just to workaround TCP behaviour (the way TCP was designed to stay alive 
and be network-friendly with re-transmit timers in tens of minutes or even 
hours and basically never timeot) which might make sense for some 
applications, but is not really what the user needs in case of interactive 
application using TCP.

> For instance, to me it would be really useful to see some explanation 
> how relavant TCP stacks deal with hibernation, and how suboptimal their 
> timers are in reality. Also, potential guidance in that space has to be 
> realistic, i. e., it must take into account the actual constraints 
> inside a real TCP stack. So, we have to understand these constraints.

http://www.pcvr.nl/tcpip/tcp_time.htm was one document I found discussing 
this, claiming in this particular test that 64 seconds is the maximum 
retransmission timer. I'll do some tests and see if my ubuntu machine has 
approximately the same value. When waiting after a resume this 64 seconds 
feels like an eternity anyway :P

---

From: Yuchung Cheng <[email-masked]>
Date: Wed, 2 Jan 2013 09:40:22 -0800

You might find this RFC relevant.
http://tools.ietf.org/html/rfc5482

On Wed, Jan 2, 2013 at 5:06 AM, Mikael Abrahamsson <[email-masked]> wrote:

> when I resume my Ubuntu 12.04 machine (and the behaviour has been identical
> since 8.04 or 8.10 when I first started running ubuntu on this laptop), I
> have to kill my ssh client because TCP is "stuck". The other end has long
> ago closed its end of the session, but my end hasn't noticed and even if I
> wait several minutes, it won't.
>
>  For instance, to me it would be really useful to see some explanation how
>> relavant TCP stacks deal with hibernation, and how suboptimal their timers
>> are in reality. Also, potential guidance in that space has to be realistic,
>> i. e., it must take into account the actual constraints inside a real TCP
>> stack. So, we have to understand these constraints.
>>
>
> http://www.pcvr.nl/tcpip/tcp_**time.htm<http://www.pcvr.nl/tcpip/tcp_time.htm>was one document I found discussing this, claiming in this particular test
> that 64 seconds is the maximum retransmission timer. I'll do some tests and
> see if my ubuntu machine has approximately the same value. When waiting
> after a resume this 64 seconds feels like an eternity anyway :P
>
>
> --
> ______________________________**_________________
> tcpm mailing list
> [email-masked]
> https://www.ietf.org/mailman/**listinfo/tcpm<https://www.ietf.org/mailman/listinfo/tcpm>
>

---

From: Mikael Abrahamsson <[email-masked]>
Date: Wed, 2 Jan 2013 18:55:27 +0100 (CET)

On Wed, 2 Jan 2013, Yuchung Cheng wrote:

Yes, it's relevant and it would help this very specific use-case if SSH 
implemented RFC5482.

However, I still feel that the overall problem would be better adressed by 
the TCP stack being able to understand that "network connectivity was 
down, now it's up again, time to wake up all connections and see if 
they're still valid" and not treat each TCP connection like ships in the 
night. I imagine this would be triggered by for instance a connection 
manager re-connecting to a network and determining that it seems to have 
full connectivity, then it would signal the TCP stack to take action on 
all existing TCP connections.

I seem to have problem even getting agreement about my problem statement, 
am I really that far off, none of you here see the same problem I do?

---

From: Joe Touch <[email-masked]>
Date: Wed, 02 Jan 2013 10:37:54 -0800

Hi, all,

FWIW:

- TCP has a "keepalive" setting, which (if enabled) will kill a 
connection whose interface is down

- in the absence of that setting, it is a key "feature" of TCP that a 
connection can be idle for an arbitrary period of time (if there is no 
outstanding unack'd data) without any packet exchanges

- TCP state isn't generally 'cleaned up' until it interferes with a new 
connection; it is at that time that the old connection state is discarded

So I'm not clear on what "stuck" means for a TCP connection in this 
context. If host connectivity is lost and re-established later, the 
connection can resume.

PS - the IETF has claimed in the past that it doesn't standardize OS 
interfaces (I agree), but IMO the result is that they too often ignore a 
critical interface of a protocol. E.g., TCP doesn't define a POSIX 
socket interface, but does define a set of commands as interface to the 
upper layer - OPEN, CLOSE, LISTEN, etc. (which are similar to the socket 
interface, but not identical). Such interfaces are critical and help 
define the protocol's semantics; it's not appropriate to assume they can 
be changed without impact.

Joe

---

From: Mikael Abrahamsson <[email-masked]>
Date: Wed, 2 Jan 2013 20:14:31 +0100 (CET)

On Wed, 2 Jan 2013, Joe Touch wrote:

> So I'm not clear on what "stuck" means for a TCP connection in this context. 
> If host connectivity is lost and re-established later, the connection can 
> resume.

Yes, but how long does it take for it to resume? I want my TCP sessions to 
be either reset (if they're not up at both ends) or taking traffic (which 
means they're not in 64s retransmit timer mode) when my connection manager 
has deemed that my network connectivity is up and running again.

"Happy Eyeballs" was invented to remove a lot of interactive user delay 
when it came to dysfunctioning IPv4 or IPv6. I would like to see similar 
work done for TCP when it comes to network connectivity being 
re-established. For my interactive sessions, I don't want to wait up to 
64s for it to decide if the session is down at the other end or not after 
a known outage (suspend/hibernate or other network outage that can be 
detected by the host).

---

From: Joe Touch <[email-masked]>
Date: Wed, 02 Jan 2013 11:20:20 -0800

On 1/2/2013 11:14 AM, Mikael Abrahamsson wrote:
>> - TCP state isn't generally 'cleaned up' until it interferes with a
>> new connection; it is at that time that the old connection state is
>> discarded
>>
> Yes, but how long does it take for it to resume? I want my TCP sessions
> to be either reset (if they're not up at both ends) or taking traffic
> (which means they're not in 64s retransmit timer mode) when my
> connection manager has deemed that my network connectivity is up and
> running again.

Why not just set the TCP to "keepalive" and set the interval you want? 
Why rely on a separate mechanism to do this when it's already there?

> re-established.

It's already there for apps that want it. Note that some apps either 
don't care or don't want this sort of behavior, though.

 > For my interactive sessions, I don't want to wait up to
> 64s for it to decide if the session is down at the other end or not
> after a known outage (suspend/hibernate or other network outage that can
> be detected by the host).

See above ;-)

Joe

---

From: David Borman <[email-masked]>
Date: Wed, 2 Jan 2013 19:24:11 +0000

On Jan 2, 2013, at 7:06 AM, Mikael Abrahamsson <[email-masked]> wrote:

> Well, as I stated in my original email, I have this problem all the time when I resume my Ubuntu 12.04 machine (and the behaviour has been identical since 8.04 or 8.10 when I first started running ubuntu on this laptop), I have to kill my ssh client because TCP is "stuck". The other end has long ago closed its end of the session, but my end hasn't noticed and even if I wait several minutes, it won't.

There has to be unacknowledged data for TCP to notice that the other side has gone away.  If everything is working properly, when you resume your machine, in an interactive SSH session if you type new data, TCP should send that off, and the other side should send back an immediate TCP reset because the connection no longer exists on the other side.  However, if no reset is received, the TCP connection will go through the normal TCP exponential back-off and retransmits until it gives up and drops the connection.  The primary reasons why you wouldn't get the reset is that the remote host is down or unreachable, or some middle box in the path is silently dropping the packets because it no longer has, or never had, state information for that connection.  Keep in mind that even if TCP gets an ICMP Host Unreachable message, it'll typically treat that as a soft error and continue trying to send the data until it times out, then when it eventually does time out, it'll report it as a host unreachable instead of timed out.

Of course, if there is no data to send, TCP won't discover that the connection is dead.  That's exactly the scenario why people added Keep-Alive timers into TCP, to allow servers to have a way to clean up idle connections where the remote side has silently gone away.

> 
> My experience is that Windows (XP and 7) in case of resume will just kill all TCP sessions, which I don't think is optimal either.
> 
> I know several applications that run their own keep-alive on top of TCP just to workaround TCP behaviour (the way TCP was designed to stay alive and be network-friendly with re-transmit timers in tens of minutes or even hours and basically never timeot) which might make sense for some applications, but is not really what the user needs in case of interactive application using TCP.

TCP can't, by itself, make decisions about when to drop connections, other than timing out a connection due to lack of response to new data, or other situations defined in the protocol.  Beyond that requires knowledge beyond the scope of TCP; e.g. when the application sets the TCP keep-alive timer, it is giving TCP additional information about how to handle particular idle connections.  Ultimately, if the application doesn't like how TCP handles things, it must implement its own method of detecting and dropping connections that have become non-responsive within the parameters that the application thinks are reasonable.

		-David Borman

> http://www.pcvr.nl/tcpip/tcp_time.htm was one document I found discussing this, claiming in this particular test that 64 seconds is the maximum retransmission timer. I'll do some tests and see if my ubuntu machine has approximately the same value. When waiting after a resume this 64 seconds feels like an eternity anyway :P
> 
> -- 

----------------------------------------------------------------------
The information contained in this transmission may be confidential. Any disclosure, copying, or further distribution of confidential information is not permitted unless such privilege is explicitly granted in writing by Quantum. Quantum reserves the right to have electronic communications, including email and attachments, sent across its networks filtered through anti virus and spam software programs and retain such messages in order to comply with applicable data security and retention requirements. Quantum is not responsible for the proper and complete transmission of the substance of this communication or for any delay in its receipt.

---

From: Mikael Abrahamsson <[email-masked]>
Date: Wed, 2 Jan 2013 20:29:36 +0100 (CET)

> Why rely on a separate mechanism to do this when it's already there?

If I set TCP keepalive to 5 seconds, I'm going to unneccessarily send a 
packet every 5 seconds for the duration of the session, right? Is that 
really a sane recommendation for solving resume problem generally?

---

From: Mikael Abrahamsson <[email-masked]>
Date: Wed, 2 Jan 2013 20:39:55 +0100 (CET)

On Wed, 2 Jan 2013, David Borman wrote:

> There has to be unacknowledged data for TCP to notice that the other 
> side has gone away.  If everything is working properly, when you resume 
> your machine, in an interactive SSH session if you type new data, TCP 
> should send that off, and the other side should send back an immediate 
> TCP reset because the connection no longer exists on the other side.

Yes, I understand that. However, if I happen to press enter in my ssh 
session before the connection manager is done with setting up my wifi, 
it's now in backoff and it seems to take a substantial amount of time 
before it recovers. Enough that I've given up on waiting and just kill it 
on reconnect. If I wait until the connection manager is done and then 
press enter, I get the RST quickly just as you describe.

What I want to happen (or that is the answer I have come up with), is 
basically when the connection manager has deemed that connectivity is up, 
it tells the TCP stack to fire all connections (send a keepalive or 
whatever), figure out what connections are still valid and reset the ones 
it receives RST to. Basically do a "state refresh". If it doesn't receive 
a response, then that session should after a while go back to 64s 
retransmit timer just like we do today.

> However, if no reset is received, the TCP connection will go through the 
> normal TCP exponential back-off and retransmits until it gives up and 
> drops the connection.  The primary reasons why you wouldn't get the 
> reset is that the remote host is down or unreachable, or some middle box 
> in the path is silently dropping the packets because it no longer has, 
> or never had, state information for that connection.  Keep in mind that 
> even if TCP gets an ICMP Host Unreachable message, it'll typically treat 
> that as a soft error and continue trying to send the data until it times 
> out, then when it eventually does time out, it'll report it as a host 
> unreachable instead of timed out.

In my case, there is no stateful inspection firewall that might timeout 
the session, yet I still see my described problem.

> Of course, if there is no data to send, TCP won't discover that the 
> connection is dead.  That's exactly the scenario why people added 
> Keep-Alive timers into TCP, to allow servers to have a way to clean up 
> idle connections where the remote side has silently gone away.

Yes, so what I want is to do "spurious keepalive" on all TCP sessions (or 
send null data) without having to put all my sessions in 5s keepalive to 
handle the two times per day I have my laptop suspended.

> TCP can't, by itself, make decisions about when to drop connections, 
> other than timing out a connection due to lack of response to new data,

I am not talking about lack of response.

> or other situations defined in the protocol.  Beyond that requires 
> knowledge beyond the scope of TCP; e.g. when the application sets the 
> TCP keep-alive timer, it is giving TCP additional information about how 
> to handle particular idle connections.  Ultimately, if the application 
> doesn't like how TCP handles things, it must implement its own method of 
> detecting and dropping connections that have become non-responsive 
> within the parameters that the application thinks are reasonable.

So what you're saying is that you don't see any use in my proposal to 
handle re-establishment of network connectivity differently from existing 
functionality already there.

---

From: Joe Touch <[email-masked]>
Date: Wed, 02 Jan 2013 11:42:27 -0800

> really a sane recommendation for solving resume problem generally?

5 seconds is a bit short for a keepalive, but it's the same packet 
whether TCP sends it or the app does. I.e., if 5 seconds is the 
keepalive you want, then there's no reason not to use TCP's internal 
mechanism for this.

Joe

---

From: Mikael Abrahamsson <[email-masked]>
Date: Wed, 2 Jan 2013 21:08:55 +0100 (CET)

> keepalive you want, then there's no reason not to use TCP's internal 

Well, it's not the behaviour I want. I want keepalives at a single 
instance, once each time my network connectivity has been re-established.

If an OS vendor implemented this without any further standardization, 
would that violate any RFC as they stand today? My thought was always that 
it would be better to have "TCP people" give guidance on this topic, but 
perhaps that is not needed (and desired).

---

From: David Borman <[email-masked]>
Date: Wed, 2 Jan 2013 14:28:43 -0600

> On Wed, 2 Jan 2013, David Borman wrote:
...
> Yes, so what I want is to do "spurious keepalive" on all TCP sessions (or send null data) without having to put all my sessions in 5s keepalive to handle the two times per day I have my laptop suspended.

I see no problem with that, basically a keep-alive that is triggered on network state transitions, rather than on a time-based interval.  FYI, TCP does not have explicit keep alive packets, it is not part of the protocol.  TCP keepalives are just carefully constructed TCP packets designed to elicit an immediate ACK from the remote side.

> 
>> TCP can't, by itself, make decisions about when to drop connections, other than timing out a connection due to lack of response to new data,
> 
>> or other situations defined in the protocol.  Beyond that requires knowledge beyond the scope of TCP; e.g. when the application sets the TCP keep-alive timer, it is giving TCP additional information about how to handle particular idle connections.  Ultimately, if the application doesn't like how TCP handles things, it must implement its own method of detecting and dropping connections that have become non-responsive within the parameters that the application thinks are reasonable.
> 
> So what you're saying is that you don't see any use in my proposal to handle re-establishment of network connectivity differently from existing functionality already there.

Nope, I didn't say that.  After a sleep/resume cycle, maybe things have changed, maybe not.  There's nothing wrong with telling TCP that things might have changed.  The question is what should TCP do with that information, beyond what it is already doing?  In the ssh example, TCP could aggressively determine whether or not particular connections are still useful, and drop them quickly.  Or not.  If I unplug my laptop from the ethernet and walk over to another office for a few minutes, and there connect via wi-fi (different network, different IP address), then do I really want all my idle SSH connections to go away?  Not necessarily, I'm going to go back to my office, plug it back into the ethernet, and then the idle SSH connections can work again when the old IP address again is available on my ethernet interface.

My point is that what may be the correct behavior in one situation, may be exactly the wrong behavior in another situation.  That's why things like TCP Keep-Alives are configurable and off by default.  The default behavior for TCP should always be to do its best to keep connections alive, no matter how much the ground shifts underneath its feet, unless told to do otherwise by the application.  And the application should also have a way for the user to tell the application what behavior should be used.

			-David Borman

---

From: Jakob Heitz <[email-masked]>
Date: Wed, 2 Jan 2013 20:48:49 +0000

> Why rely on a separate mechanism to do this when it's already there?

Even if you set the keepalive to 5 seconds, it takes an eon
of retransmissions before the session actually goes down.

This is why BGP (at least) has a user level keepalive.

-- 
Jakob Heitz.

---

From: Jakob Heitz <[email-masked]>
Date: Wed, 2 Jan 2013 20:56:26 +0000

On , Joe Touch <> wrote:

> interfaces (I agree)

http://tools.ietf.org/html/rfc3493
http://tools.ietf.org/html/rfc3542

are examples of when they do.
These RFCs are referenced in code, even though they are informational.
I find them useful.

-- 
Jakob Heitz.

---

From: Mikael Abrahamsson <[email-masked]>
Date: Wed, 2 Jan 2013 21:57:01 +0100 (CET)

On Wed, 2 Jan 2013, David Borman wrote:

> Nope, I didn't say that.  After a sleep/resume cycle, maybe things have 
> changed, maybe not.  There's nothing wrong with telling TCP that things 
> might have changed.  The question is what should TCP do with that 
> information, beyond what it is already doing?  In the ssh example, TCP 
> could aggressively determine whether or not particular connections are 
> still useful, and drop them quickly.  Or not.  If I unplug my laptop 
> from the ethernet and walk over to another office for a few minutes, and 
> there connect via wi-fi (different network, different IP address), then 
> do I really want all my idle SSH connections to go away?  Not 
> necessarily, I'm going to go back to my office, plug it back into the 
> ethernet, and then the idle SSH connections can work again when the old 
> IP address again is available on my ethernet interface.

My idea the whole time has been that this would be configurable in the 
conncetion manager. I just wanted an API (or functionality) for the 
program to trigger the TCP stack functionality I am describing, and I 
thought it would be a good idea to have a guiding document that describes 
this.

> My point is that what may be the correct behavior in one situation, may 
> be exactly the wrong behavior in another situation.  That's why things 
> like TCP Keep-Alives are configurable and off by default.  The default 
> behavior for TCP should always be to do its best to keep connections 
> alive, no matter how much the ground shifts underneath its feet, unless 
> told to do otherwise by the application.  And the application should 
> also have a way for the user to tell the application what behavior 
> should be used.

For me, doing this on an application basis is fine in some scenarios, and 
in some I just want this to be a system wide thing. If I know that a 
change of subnetwork/IP address is permanent and I don't want any 
lingering TCP sessions bound to old IPs, I would like to configure the 
connection manager to tell the TCP stack to just close all TCP sessions 
bound to the old IPs. This was an API I also thought would make sense to 
have a document describing (or at least describe the high-level 
functionality).

---

From: Joe Touch <[email-masked]>
Date: Wed, 02 Jan 2013 13:03:39 -0800

> are examples of when they do.

RFC 793 has an API as well.

> These RFCs are referenced in code, even though they are informational.

IMO, they should be more than informational, though.

Joe

---

From: Yuchung Cheng <[email-masked]>
Date: Wed, 2 Jan 2013 13:20:10 -0800

Hi Mikael,

On Wed, Jan 2, 2013 at 12:57 PM, Mikael Abrahamsson <[email-masked]> wrote:
>> information, beyond what it is already doing?  In the ssh example, TCP could
>> aggressively determine whether or not particular connections are still
>> useful, and drop them quickly.  Or not.  If I unplug my laptop from the
>> ethernet and walk over to another office for a few minutes, and there
>> connect via wi-fi (different network, different IP address), then do I
>> really want all my idle SSH connections to go away?  Not necessarily, I'm
>> going to go back to my office, plug it back into the ethernet, and then the
>> idle SSH connections can work again when the old IP address again is
>> available on my ethernet interface.
>
>
> conncetion manager. I just wanted an API (or functionality) for the program
> to trigger the TCP stack functionality I am describing, and I thought it
> would be a good idea to have a guiding document that describes this.

FWIW, we are developing/experimenting a "system" similar to your
connection manager idea to guide TCP connections.
http://www.ietf.org/proceedings/84/slides/slides-84-iccrg-3.pdf

Dave and Joe are discussing that such a system can interface with TCP,
not built into TCP itself. The TCP people see such a system as
separate component, but app developers consider this part of
transport/network layer or sometimes kernel's job.

>
>
>> My point is that what may be the correct behavior in one situation, may be
>> exactly the wrong behavior in another situation.  That's why things like TCP
>> Keep-Alives are configurable and off by default.  The default behavior for
>> TCP should always be to do its best to keep connections alive, no matter how
>> much the ground shifts underneath its feet, unless told to do otherwise by
>> the application.  And the application should also have a way for the user to
>> tell the application what behavior should be used.
>
>
> For me, doing this on an application basis is fine in some scenarios, and in
> some I just want this to be a system wide thing. If I know that a change of
> subnetwork/IP address is permanent and I don't want any lingering TCP
> sessions bound to old IPs, I would like to configure the connection manager
> to tell the TCP stack to just close all TCP sessions bound to the old IPs.
> This was an API I also thought would make sense to have a document
> describing (or at least describe the high-level functionality).
>
>
> --

---

From: Joe Touch <[email-masked]>
Date: Wed, 02 Jan 2013 13:21:54 -0800

> instance, once each time my network connectivity has been re-established.

How do you know? Network connectivity is defined as a connected path, 
not just an interface state.

PS - if you have keepalives set, TCP doesn't necessarily send keepalive 
packets if there are already other data packets being exchanged. The 
packets are sent only when nothing else is.

> would that violate any RFC as they stand today? My thought was always
> that it would be better to have "TCP people" give guidance on this
> topic, but perhaps that is not needed (and desired).

OS vendors already have implemented this. However, if you want a timer 
that flushes all current connections when a TCP goes away that's fine 
too - just note that it may kill connections you want. The user level - 
whether implemented inside the OS, in a library, in user code - can 
always ABORT or CLOSE connections any time it wants.

Joe

---

From: Mikael Abrahamsson <[email-masked]>
Date: Thu, 3 Jan 2013 09:23:33 +0100 (CET)

> http://www.ietf.org/proceedings/84/slides/slides-84-iccrg-3.pdf

That looks very promising. To me, it sounds like the functionality I'm 
looking for could be incorporated in a system/framework you're describing 
in those slides. The system you're describing would be helped to get 
information from connection manager, as the connection manager actually 
knows a little bit about what connection is currently handling default 
route traffic (if the computer is moved from wifi, to mobile Internet, to 
fixed connection) and the least your system would need then I guess, would 
be to get notified that things have changed and take into account that 
network connectivity has changed.

If the connection manager knows that the 3G connection is down and if it's 
coming back up, it's for sure that the old IP address isn't coming back, 
then it would make sense to just reset all TCP sessions associated with 
that address.

So I believe the connection manager has valuable information for TCP, or 
something that interfaces with TCP and instructs TCP what to do. So 
perhaps this actually is not a TCP problem per se, apart from that there 
is need for an API to control TCP.

> not built into TCP itself. The TCP people see such a system as separate 
> component, but app developers consider this part of transport/network 
> layer or sometimes kernel's job.

I would agree that the view of such a system might differ from who you 
ask.

---

From: Mikael Abrahamsson <[email-masked]>
Date: Thu, 3 Jan 2013 09:26:30 +0100 (CET)

On Wed, 2 Jan 2013, Joe Touch wrote:

> How do you know? Network connectivity is defined as a connected path, not 
> just an interface state.

I am well aware of that, and I don't want to send keepalives every 5 
seconds during my entire uptime just because I want to solve a 
suspend/resume issue.

> packets are sent only when nothing else is.

I am aware of that.

> always ABORT or CLOSE connections any time it wants.

I don't what wholesale killing of all connections the way Windows seem to 
do it. I want the sessions which are still viable to be detected as such, 
the ones that are not viable (other end sends RST) should go away asap.

---

From: Olivier Bonaventure <[email-masked]>
Date: Thu, 03 Jan 2013 11:28:12 +0100

Mikael,
>
>> Nope, I didn't say that.  After a sleep/resume cycle, maybe things
>> have changed, maybe not.  There's nothing wrong with telling TCP that
>> things might have changed.  The question is what should TCP do with
>> that information, beyond what it is already doing?  In the ssh
>> example, TCP could aggressively determine whether or not particular
>> connections are still useful, and drop them quickly.  Or not.  If I
>> unplug my laptop from the ethernet and walk over to another office for
>> a few minutes, and there connect via wi-fi (different network,
>> different IP address), then do I really want all my idle SSH
>> connections to go away?  Not necessarily, I'm going to go back to my
>> office, plug it back into the ethernet, and then the idle SSH
>> connections can work again when the old IP address again is available
>> on my ethernet interface.
>
> thought it would be a good idea to have a guiding document that
> describes this.

Multipath TCP already includes such functionnalities and this is 
transparent for the application. You can start an ssh when plugged over 
ethernet, move to a pub and multipath TCP will open a subflow over the 
WiFi interface. When you come back to the office, a new subflow is open 
automatically and your ssh session continues.

The Multipath TCP code in the Linux kernel
( http://www.multipath-tcp.org ) already supports this

Olivier

-- 
INL, ICTEAM, UCLouvain, Belgium, http://inl.info.ucl.ac.be

---

From: Mikael Abrahamsson <[email-masked]>
Date: Thu, 3 Jan 2013 12:40:41 +0100 (CET)

On Thu, 3 Jan 2013, Olivier Bonaventure wrote:

> Multipath TCP already includes such functionnalities and this is transparent 
> for the application.

The application doesn't know anything about whether the ethernet plug is 
in or not, the connection manager does.

> You can start an ssh when plugged over ethernet, move to a pub and 
> multipath TCP will open a subflow over the WiFi interface. When you come 
> back to the office, a new subflow is open automatically and your ssh 
> session continues.

How long does that take? Remember network might have been out for 5 
minutes so all sessions might be in max retransmit timer mode.

Also, how does multipath TCP handle multiple default routes? If I have 
wifi and 4G and LAN at the same time, the OS usually selects the LAN 
default route for all traffic, meaning Wifi and 4G IPs will be dropped by 
ISP antispoofing filter.

---

From: Olivier Bonaventure <[email-masked]>
Date: Thu, 03 Jan 2013 14:07:00 +0100

Mikael,
>> transparent for the application.
>

Multipath TCP is informed about the arrival/removal of IP addresses and 
can create the necessary subflows.

>> multipath TCP will open a subflow over the WiFi interface. When you
>> come back to the office, a new subflow is open automatically and your
>> ssh session continues.
>

Standard TCP timer setting applies if there are retransmissions. A 
Multipath TCP connection is composed of one of more TCP subflows. 
Multipath TCP schedules the packet transmission on the different 
available subflows. If no subflow is available, Multipath TCP can create 
a new one. A TCP subflow may die due to excessive retransmissions but 
the Multipath TCP connection state could potentially remain active and 
another subflow could be reestablished.

> default route for all traffic, meaning Wifi and 4G IPs will be dropped
> by ISP antispoofing filter.

No, Multipath TCP works by using one TCP subflow over each interface. If 
you have LAN, 4G and WiFi, you have 3 IP addresses. Let's assume that 
the LAN address is the default one. If you create an MPTCP connection, 
it will be initiated from this address. Once the three-way handshake has 
been done over the LAN interface, MPTCP will start a new handshake over 
the 4G and over the WiFi interface. You then have three subflows that 
can be used to send and receive data. If you lose LAN interface or 
change its IP address, a new subflow can be reestablished.

You can find some information about the design of MPTCP at

http://inl.info.ucl.ac.be/publications/how-hard-can-it-be-designing-and-implementing-deployable-multipath-tcp

and its usage in mobile environments at

http://inl.info.ucl.ac.be/publications/exploring-mobilewifi-handover-multipath-tcp

Olivier

---

From: Mikael Abrahamsson <[email-masked]>
Date: Thu, 3 Jan 2013 14:39:37 +0100 (CET)

> change its IP address, a new subflow can be reestablished.

You seem to have misunderstood my question.

If I have eth0, ppp0 and wlan0 with one address each, my system will still 
only have a single default route, where all packets are going out. I am 
not allowed to source traffic from ppp0 and wlan0 out eth0 as it will be 
dropped by the ISP router as soon as it reaches their router (uRPF).

So unless my system also uses source based routing and/or has each 
interface in its own routing domain, even though I now have three IP 
addresses, only the eth0 one is actually usable for reaching The Internet 
(as any traffic sourced from ppp0 and wlan0 will be dropped).

> http://inl.info.ucl.ac.be/publications/how-hard-can-it-be-designing-and-implementing-deployable-multipath-tcp

A quick browse yielded nothing about my above question.

---

From: Olivier Bonaventure <[email-masked]>
Date: Thu, 03 Jan 2013 14:47:41 +0100

Mikael,
>
>> No, Multipath TCP works by using one TCP subflow over each interface.
>> If you have LAN, 4G and WiFi, you have 3 IP addresses. Let's assume
>> that the LAN address is the default one. If you create an MPTCP
>> connection, it will be initiated from this address. Once the three-way
>> handshake has been done over the LAN interface, MPTCP will start a new
>> handshake over the 4G and over the WiFi interface. You then have three
>> subflows that can be used to send and receive data. If you lose LAN
>> interface or change its IP address, a new subflow can be reestablished.
>
> If I have eth0, ppp0 and wlan0 with one address each, my system will
> still only have a single default route, where all packets are going out.
> I am not allowed to source traffic from ppp0 and wlan0 out eth0 as it
> will be dropped by the ISP router as soon as it reaches their router
> (uRPF).

Of course. Multipath TCP will only send packets with the IP address 
assigned to each interface when sending them over one interface. We know 
that packet with other source addresses will be subject to uRPF.

> addresses, only the eth0 one is actually usable for reaching The
> Internet (as any traffic sourced from ppp0 and wlan0 will be dropped).
>

The Linux implementation uses one routing table per interface so that in 
your case the ppp0 and wlan0 interfaces can be used as well.

See

http://mptcp.info.ucl.ac.be/pmwiki.php?n=Users.ConfigureRouting

Olivier

---

From: Mikael Abrahamsson <[email-masked]>
Date: Thu, 3 Jan 2013 14:58:39 +0100 (CET)

On Thu, 3 Jan 2013, Olivier Bonaventure wrote:

> The Linux implementation uses one routing table per interface so that in your 
> case the ppp0 and wlan0 interfaces can be used as well.
>
> See
>

Exactly what I was looking for, thanks. I've seen the same problem with 
SCTP implementations that didn't have this as well. Vendors are getting 
better at this over time though.

What we need to happen is that the connection manager should do this 
automatically as it adds routes etc. This needs to work dynamically with 
DHCP and other ways of handing out IPs and routes. Have you had any luck 
in getting traction for this with OS vendors?

---

From: Joe Touch <[email-masked]>
Date: Thu, 03 Jan 2013 13:35:45 -0800

On 1/3/2013 12:26 AM, Mikael Abrahamsson wrote:
>> PS - if you have keepalives set, TCP doesn't necessarily send
>> keepalive packets if there are already other data packets being
>> exchanged. The packets are sent only when nothing else is.
>
> I am aware of that.
>
>> too - just note that it may kill connections you want. The user level
>> - whether implemented inside the OS, in a library, in user code - can
> I don't what wholesale killing of all connections the way Windows seem
> to do it. I want the sessions which are still viable to be detected as
> such, the ones that are not viable (other end sends RST) should go away
> asap.

That's the trick; "viable" is a relative thing, at best known only when 
you actually want to send more data on the connection. That's when most 
TCP implementations bother to clean up stale state, since that's the 
only time it really matters.

Joe

---

From: Mikael Abrahamsson <[email-masked]>
Date: Fri, 4 Jan 2013 07:16:20 +0100 (CET)

> only time it really matters.

Well, there is data outbound I want to send it *ASAP*, but TCP thinks it 
needs to wait 50 seconds because it's been in exponential backoff because 
of the network outage that is now rectified (and known by the connection 
manager as being rectified).

Why should I wait 49 seconds extra because the connection manager and TCP 
doesn't share information?

The connection manager also knows that half my TCP sessions are never 
going to be viable ever again because the IP address they're bound to is 
gone and is never going to return (because it was on my 4G USB dongle).

---

From: Joe Touch <[email-masked]>
Date: Thu, 03 Jan 2013 22:46:59 -0800

On 1/3/2013 10:16 PM, Mikael Abrahamsson wrote:
>> That's the trick; "viable" is a relative thing, at best known only
>> when you actually want to send more data on the connection. That's
>> when most TCP implementations bother to clean up stale state, since
>> that's the only time it really matters.
>
> needs to wait 50 seconds because it's been in exponential backoff
> because of the network outage that is now rectified (and known by the
> connection manager as being rectified).
>
> Why should I wait 49 seconds extra because the connection manager and
> TCP doesn't share information?

The trouble is that there are other cases that are indistinguishable - 
i.e., when the network interface goes down then up but the reason 
packets aren't getting through is that the network is congested.

I.e., sharing information isn't the issue; it's the fact that the 
information your sharing isn't complete. Did your connection manager 
check the path to make sure that the TCP packets will get through? Did 
it do probes in a way that won't create congestion in a case where 
congestion was what brought the path down (or made it appear so)? That's 
exactly why TCP does what it does. Assuming that some omniscient entity 
knows better is a fine academic exercise, but easily creates a tragedy 
of the commons in the real world.

> gone and is never going to return (because it was on my 4G USB dongle).

How does it "know" the address will never return? Restarting your dongle 
could easily result in reassigning the same address.

Joe

---

From: Jakob Heitz <[email-masked]>
Date: Fri, 4 Jan 2013 07:36:43 +0000

The trouble with TCP keepalives is they take too long.
1. They should not be sent more frequently than 2 hours.
2. If the keepalive fails, it takes at least 100 seconds to find out.

TCP needs an "on demand" keepalive.
For example, send 3 of them, one second apart and
die if no response in 5 seconds. Something like that.

On , Joe Touch <> wrote:

>> Well, there is data outbound I want to send it *ASAP*, but TCP
>> thinks it needs to wait 50 seconds because it's been in exponential
>> backoff because of the network outage that is now rectified (and
>> known by the connection manager as being rectified).
>> 
>> going to be viable ever again because the IP address they're bound
>> to is gone and is never going to return (because it was on my 4G USB
>> dongle). 
> 

-- 
Jakob Heitz.

---

From: Joe Touch <[email-masked]>
Date: Fri, 04 Jan 2013 10:55:53 -0800

Please review RFC 1122, Section 4.2.3.6

Keepalives *default* to off and a 2-hour interval, but also *must* be 
configurable.

If you want a 1-second keepalive, you should configure it as such.

Joe

---

From: Jakob Heitz <[email-masked]>
Date: Fri, 4 Jan 2013 19:04:18 +0000

> If you want a 1-second keepalive, you should configure it as such.

Also in TCP, if the keepalive fails, it takes an
eternity of retransmissions before TCP fails.

This is why many applications, including BGP,
use an application level keepalive.

I don't want a 1 second keepalive.
I want an on-demand keepalive.

-- 
Jakob Heitz.

---

From: "Scharf, Michael (Michael)" <[email-masked]>
Date: Fri, 4 Jan 2013 20:44:19 +0100

> Behalf Of Jakob Heitz
> Sent: Friday, January 04, 2013 8:04 PM
> To: Joe Touch
> Cc: [email-masked]
> Also in TCP, if the keepalive fails, it takes an eternity of 
> retransmissions before TCP fails.
> 
> This is why many applications, including BGP, use an 
> application level keepalive.
> 

Maybe a very naive question: What does prevent the app from reducing the keep-alive interval "on-demand"?

At first sight, RFC 1122 does not mandate that the interval has to be constant.

Michael

---

From: Joe Touch <[email-masked]>
Date: Fri, 04 Jan 2013 11:51:48 -0800

> At first sight, RFC 1122 does not mandate that the interval has to be constant.

Right - that was my thought.

If what you want is app-controlled keepalive, then implement it at the 
app layer. But so far what I see is an optimization that might work in 
some cases, but could backfire in a lot of others.

Joe

---

From: Joe Touch <[email-masked]>
Date: Fri, 04 Jan 2013 11:56:19 -0800

> eternity of retransmissions before TCP fails.

There are several keepalive parameters:

	time - duration between two keepalive tests; defaults to 2 hours
		but must be configurable

	interval - duration between successive keepalive transmissions

	retry - how many transmissions fail before giving up

Set these as you want.

> use an application level keepalive.

BGP uses a keepalive because it incorrectly links TCP connectivity to IP 
connectivity - i.e., if TCP goes down, it wants to pull routes to that 
endpoint (which was a mistake IMO).

So it has to try to keep the TCP connection up. It uses its own 
keepalives because keepalive isn't required at the TCP layer; these days 
it could easily use TCP keepalives instead.

Joe

---

From: Ethan Blanton <[email-masked]>
Date: Mon, 7 Jan 2013 22:51:37 -0500

Mikael Abrahamsson spake unto us the following wisdom:
> >So I'm not clear on what "stuck" means for a TCP connection in
> >this context. If host connectivity is lost and re-established
> >later, the connection can resume.
> 
> Yes, but how long does it take for it to resume? I want my TCP
> sessions to be either reset (if they're not up at both ends) or
> taking traffic (which means they're not in 64s retransmit timer
> mode) when my connection manager has deemed that my network
> connectivity is up and running again.

I actually think this is a more complex problem, caused by middle
boxes and NATs and "security" and all kinds of things.

When your PC resumes and you go to that SSH session and hit Enter,
that will trigger SSH to send some data, and a TCP segment will go out
almost immediately.  (That is, assuming there wasn't data being
actively sent at the time the machine went to sleep, which is often a
valid assumption.)  This will cause the other end to either ACK or
RST, in a perfect world, at which point you'll either have a usable
connection or you'll find out you were disconnected.

Unfortunately, the world isn't perfect, and a LARGE percentage of the
time, that data segment just disappears into the Ether.  I'm not sure
what all of the reasons for that are, but here are a couple of
well-known reasons:

 * Stateful NATs and/or firewalls with timeouts can "forget" about
   your TCP connection whilst your machine is asleep, then silently
   drop your data segment because they "didn't see a SYN" and don't
   have state for your connection.

 * Misguided "security" policies on the remote machine may cause them
   to fail to reset packets from unknown connections, if the other
   machine had been sending data while your machine was asleep and
   eventually timed out.

The trouble with these reasons is that they're not necessarily
something you can fix by telling TCP "hey, shorten your timeouts",
unless you want to make them *way* short (you're unhappy with ~1
minute, apparently?).  Since the outgoing packets just silently
disappear with no reply, the local TCP has no way of telling if
there's a transient network issue (maybe not at the local interface,
if your system just told the TCP the interface is good), if the
end-to-end delay is simply long, etc.

This seems like something that should be handled by applications that
are specifically concerned about interactivity.  If you don't have
interactive goals, wait 64 seconds (or longer) and see what happens.
If you do, then send an application layer ping after a resume, give it
however long the user should be expected to wait, and reconnect.

Ethan

---

From: Mikael Abrahamsson <[email-masked]>
Date: Tue, 8 Jan 2013 10:32:57 +0100 (CET)

> packets aren't getting through is that the network is congested.

If the network interface goes down, packets aren't going to pass. When it 
comes back up again, there is a chance packets might pass. I don't see how 
this can be indistinguishable.

> of the commons in the real world.

The connection manager knows when the network is completely down (network 
interface down), and it knows when it might start working again. That's 
why I'm not proposing that it should tell TCP to go into fast-retransmit 
when the cable is up again and DHCP address has been negotiated and there 
now is a chance it might work again. I'm proposing retransmit numbers in 
the range of 1 or a few seconds, instead of 64.

> could easily result in reassigning the same address.

Because I have checkboxed in my 4G connection profile that addresses are 
dynamic and will be new each time there is a connection, so the connection 
manager knows because I've told it so.

---

From: Mikael Abrahamsson <[email-masked]>
Date: Tue, 8 Jan 2013 10:40:55 +0100 (CET)

On Mon, 7 Jan 2013, Ethan Blanton wrote:

> almost immediately.

No, it usually doesn't, because I pressed it too soon so the first 3-4 
transmits went to nowhere because there was no wifi up yet.

> what all of the reasons for that are, but here are a couple of 

I have none of these problems. With IPv6, I see them being less and less 
prevelant, thus the need for the functionality I am describing.

> The trouble with these reasons is that they're not necessarily something 
> you can fix by telling TCP "hey, shorten your timeouts", unless you want 
> to make them *way* short (you're unhappy with ~1 minute, apparently?). 
> Since the outgoing packets just silently disappear with no reply, the 
> local TCP has no way of telling if there's a transient network issue 
> (maybe not at the local interface, if your system just told the TCP the 
> interface is good), if the end-to-end delay is simply long, etc.

I have no problem with that. I want to fix the case where TCP can get an 
RST or ACK immediately if it just starts talking instead of waiting tens 
of seconds.

> interactive goals, wait 64 seconds (or longer) and see what happens. If 
> you do, then send an application layer ping after a resume, give it 

What should the application do exactly? And what mechanism tells the 
application that the host has resumed so the application should now tell 
TCP to do things. And why does it make more sense to have the connection 
manager communicate to all applications, instead of just communicating to 
the kernel TCP (SCTP and others) and have them figure out the problem.

I see TCP as the easy target here, instead of modifying a lot of 
applications.

---

From: Joe Touch <[email-masked]>
Date: Wed, 09 Jan 2013 15:51:07 -0800

On 1/8/2013 1:32 AM, Mikael Abrahamsson wrote:
> If the network interface goes down, packets aren't going to pass. When
> it comes back up again, there is a chance packets might pass. I don't
> see how this can be indistinguishable.

It's indistinguishable when packets don't pass but the network interface 
is up. Just because a network interface is up doesn't mean it's safe - 
or appropriate - to start resending at full rate.

>> congestion was what brought the path down (or made it appear so)?
>> That's exactly why TCP does what it does. Assuming that some
>> omniscient entity knows better is a fine academic exercise, but easily
>> creates a tragedy of the commons in the real world.
>
> The connection manager knows when the network is completely down
> (network interface down), and it knows when it might start working
> again. That's why I'm not proposing that it should tell TCP to go into
> fast-retransmit when the cable is up again and DHCP address has been
> negotiated and there now is a chance it might work again. I'm proposing
> retransmit numbers in the range of 1 or a few seconds, instead of 64.

Fast retransmit is intended when the ACKs get through but indicate that 
a segment was likely lost. Again, just because the network interface 
comes back doesn't mean that a resent segment will get through.

>> How does it "know" the address will never return? Restarting your
>> dongle could easily result in reassigning the same address.
>
> dynamic and will be new each time there is a connection, so the
> connection manager knows because I've told it so.

Dynamic addresses can still be reassigned as the same, though.

There are only three places where things happen regarding TCP:
	- inside the TCP protocol
	- at the app layer
	- on the interface (segment arrivals/departures)

If you're talking about modifying the app layer (which could be 
implemented in the OS, a library, or anywhere), then this isn't a change 
to TCP, and isn't relevant for either an RFC or this list.

If you're talking about modifying TCP for this purpose, it still seems 
very inappropriate.

Joe