Subject: RE: Comments on draft-ietf-multi6-v4-multihoming-02

From: <[email-masked]>
Date: Thu, 11 Nov 2004 14:55:17 +0200

Ilijitsch,

> > So, are you suggesting that if we want to maintain any multihoming  bindings,
> > we should rely on application layer keep alives?
> 
> Certainly not. What I'm saying is that if applications want to be sure 
> a session remains available, they should send keepalives. Then, if the 
> session goes away, the app gets to hear about it without much delay.

Got it.
 
> As far as multihoming goes, we have no business repairing onnectivity 
> that isn't used. Not only does this use up unnecessary bandwidth, it 
> also means unnecessary work as the failures we try to repair may have 
> gone away on their own when the communication resumes. There 
> is really no upside to being overzealous here.

Agree.  
 
> > So what I meant more was that if we use keep-alives at the multi6
> > layer, then perhaps this could be something that the upper layer
> > could set the keep-alive behavior.
> 
> I could just go on disagreeing, but let me ask you this: assuming a 
> decent multihoming solution, under which circumstances would upper 
> layers want to do something like that, trying to achieve which 
> benefits?

I'm not wedded to the use of keep-alives; to back it up a level,
if there are keep alives at the multi6 level, it would be nice if
the use of the keep-alive could be tunable by the ULP.  If we 
toss out keep-alives at the multi6 level, I won't lose sleep at it.
 
> > However, I still don't know if you are suggesting that we use 
> > keep-alives at the multi6 layer (in some circumstances) or if you 
> > would kick this up to the application layer.
> 
> The multihoming layer needs find at least one working address pair (if 
> it exists) from the set of all possible address pairs. If we assume 
> bidirectional connectivity this isn't all that difficult: send a 
> ping-like packet and see if something comes back. But we need to be 
> able to use two unidirectional paths rather than one bidirectional one 
> in a world where it's possible that routing and ingress filtering get 
> in each other's way. Unidirectional reachability testing is a lot 
> harder as we need the results of the reachability testing in one 
> direction in order to send back the results of reachability testing in 
> the other direction.

Ack.
 
> How is an application ever going to do anything useful in this area, 
> even assuming applications that are smart enough (which they won't be)? 
> Their interaction with the network is obscured by a transport layer 
> which generally hides any unreachability, and the multihoming layer 
> which hides address changes. It's like driving a car from inside the 
> trunk.

At the same time, how does the multi6 layer understand what kind of 
connectivity the application needs?  With wireless links, its really 
easy to get intermittent connectivity. How do we classify when the
connectivity is too poor to be useful - the application should be able
to determine this and ask for using a different address pair.

John

---

From: Joe Touch <[email-masked]>
Date: Thu, 11 Nov 2004 06:13:28 -0800

[email-masked] wrote:
...
> connectivity the application needs?

It can't unless it knows whether its TCP or UDP, which (strictly) it 
can't know either (e.g., IPsec-encrypted).

 > With wireless links, its really

If the connectivity doesn't allow a TCP connection to be used with 
little or no delay whenever the app deems it necessary, there's a problem.

----

Has anyone considered keeping the connection 'active' (i.e., keepalives) 
for sessions for around 1-2 typical RTTs of idle? That would catch TCP 
idles that still allow TCP bursts; anything longer than that and TCP is 
supposed to 'restart' its congestion control anyway.

That might be a happy medium to the 'all or none' keepalive. I.e.,
"keepalive for 1-2 RTTs of idle".

Joe

---

From: Iljitsch van Beijnum <[email-masked]>
Date: Fri, 12 Nov 2004 09:47:43 +0100

John,

> toss out keep-alives at the multi6 level, I won't lose sleep at it.

Keepalives aren't really the correct name for what the multi6 layer 
needs. The multi6 layer needs to know which address pairs work and 
which don't. There are basically two approaches to this: try every 
combination and pick the one that loooks best, or start trying 
combinations until you find the first one that works and use that one. 
(These are extremes, something in the middle is also possible.) This 
reachability testing MUST happen in the multi6 layer for robustness and 
security reasons.

The question that we've been talking about is when this reachability 
testing should happen. I don't think sending keepalives all the time or 
when idle is a good idea here. For multihoming purposes, it's enough to 
do reachability tests when the traffic patterns indiciate a _possible_ 
loss of connectivity. This would be when there is only traffic in one 
direction.

Now if applications or transports feel they need to notice immediately 
when connectivity goes away, they can always send keepalives when the 
session is otherwise idle. It doesn't make sense to subcontract this 
job to the multihoming layer for two reasons:

1. There may not be any multihoming so no keepalives happen
2. The added complexity of interfacing between layers

> to determine this and ask for using a different address pair.

This is an unsolved problem. I think at some point we need an interface 
for this, so that multi6-aware applications can influence certain 
aspects of multihoming operation. I've always assumed that once basic 
multihoming is in the can the next step would be to make TCP 
multihoming-aware so that it can load balance over more than one 
address pair at a time in order to achieve x+y throughput rather than 
max(x,y) throughput, or worse 
min(x,y)+abs(rand()+0.5)*(max(x,y)-min(x,y)) (ok this means either x or 
y at random) throughput.

---

From: Iljitsch van Beijnum <[email-masked]>
Date: Fri, 12 Nov 2004 10:01:41 +0100

On 11-nov-04, at 15:13, Joe Touch wrote:

> Has anyone considered keeping the connection 'active' (i.e., 
> keepalives) for sessions for around 1-2 typical RTTs of idle? That 
> would catch TCP idles that still allow TCP bursts; anything longer 
> than that and TCP is supposed to 'restart' its congestion control 
> anyway.

> "keepalive for 1-2 RTTs of idle".

Hm, wouldn't just one or two RTTs be a bit short? You may still get a 
delayed ACK after 2 RTTs.

I think this basically boils down to the idea I was talking about 
previously, where there are only reachability checks when there are 
packets going in one direction. So when we transmit a packet, we start 
a timer. When we receive a packet, we cancel the timer. When the timer 
expires, we test reachability. Some optimization is possible with TCP 
so that we don't start the timer for ACK packets with no data, but we 
do stop the timer for those.

---

From: Joe Touch <[email-masked]>
Date: Fri, 12 Nov 2004 07:17:33 -0800

Iljitsch van Beijnum wrote:
>> than that and TCP is supposed to 'restart' its congestion control anyway.
> 
> 

The delayed ACK should itself refresh things; I'm thinking of an 
otherwise completely idle interaction. The key here is that TCP expects 
things not to change for a few RTTs (pick a number). Designing things so 
that is not the case is likely to cause unintended impact on TCP, esp. 
in some very typical cases (web surfing over persistent connections).

> packets going in one direction. So when we transmit a packet, we start a 
> timer. When we receive a packet, we cancel the timer. When the timer 
> expires, we test reachability. Some optimization is possible with TCP so 
> that we don't start the timer for ACK packets with no data, but we do 
> stop the timer for those.

I'm suggesting that the above be tempered after a few RTTs. I.e., if the 
only traffic has been the reachability tests for more than X RTTs (pick 
X), then it might be OK to stop testing. This solves the "persistent 
chatter over idle channels" problem, and (appropriately?) allows 
underlying asspciations to be dropped/moved without substantial impact.

Joe

---

From: Iljitsch van Beijnum <[email-masked]>
Date: Sat, 13 Nov 2004 14:50:08 +0100

> otherwise completely idle interaction.

But how do you know a session is completely idle vs there is still a 
delayed ack coming? (Without looking too deep into TCP, that is.)

> The key here is that TCP expects things not to change for a few RTTs 
> (pick a number). Designing things so that is not the case is likely to 
> cause unintended impact on TCP, esp. in some very typical cases (web 
> surfing over persistent connections).

(Note that is makes little difference whether you use persistent 
connections: we work at the IP layer.)

The trick is detecting legitimate idleness because in that state no 
action is necessary. For TCP this shouldn't be too hard, for other 
protocols we have to find good timeout values. I think these need to be 
higher than just a few RTTs.

> I'm suggesting that the above be tempered after a few RTTs. I.e., if 
> the only traffic has been the reachability tests for more than X RTTs 
> (pick X), then it might be OK to stop testing.

Hm, my assumption was that after a successful test, there wouldn't be 
any more tests until something changes. Since we don't do anything when 
the session is active in both directions or completely idle, 
reachability tests only happen after a suspicious active->idle 
transition. After the test, the multihoming layer would be satisfied 
that this transition wasn't because of a failure, and wouldn't have to 
do anything until there is a new active->idle transition.

Iljitsch

---

From: Joe Touch <[email-masked]>
Date: Sat, 13 Nov 2004 19:29:36 -0800

> delayed ack coming? (Without looking too deep into TCP, that is.)

Delayed ACKs come within 200ms of the other end going idle, AFAIR.

> connections: we work at the IP layer.)

Persistent connections are the kind of TCP connections that might go 
idle for extended periods; most other TCP connections tend to either be 
sending data or be closed quickly. I.e., they're the kind of transport 
layer that would care whether you did keep-alives at the net layer, IMO.

> higher than just a few RTTs.

Such as? I was thinking 300ms-500ms as a ballpark.

> any more tests until something changes. 

That presumes async notification from the lower layer. I wasn't, since I 
don't believe that's feasible.

I.e., I would expect:
	- seeing packets exchanged is an in-band test that can be
	  observed and serves as a keep-alive itself
	- send keepalives for the first 500ms after idle
	- stop sending keepalives after that

In the absence of keepalives, things can be changed or dropped as needed 
fairly quickly, as a result.

> Since we don't do anything when 

I'd like it to do something - to keep verifying that reachability is 
still possible for exactly the period that an idle TCP 'expects' there 
not to be a substantial change in the channel.

Joe

---

From: Iljitsch van Beijnum <[email-masked]>
Date: Sun, 14 Nov 2004 18:22:33 +0100

> Delayed ACKs come within 200ms of the other end going idle, AFAIR.

Ok. But then we have to account for normal packet loss and normal RTT 
variations.

> idle for extended periods;

Actually I believe they are closed after a relatively short time when 
they aren't used. And very few sessions use persistent connections last 
time I checked.

> most other TCP connections tend to either be sending data or be closed 
> quickly.

A better example of sessions that remain open long even if they're idle 
are SSH sessions.

>> protocols we have to find good timeout values. I think these need to 
>> be higher than just a few RTTs.

> Such as? I was thinking 300ms-500ms as a ballpark.

I was thinking along the lines of 10 - 15 seconds for non-TCP stuff. I 
heaven't really looked in detail, but I'm pretty sure transports for AV 
streaming don't generate ACKs every 500 ms.

Alternatively, this could be made to auto-tune.

Note though that if you select value X for this and ACKs are sent every 
1.1*X interval, you'll be sending test packets every X. With a larger X 
this is less likely to happen, but also results in less unnecessary 
traffic if it does.

> That presumes async notification from the lower layer. I wasn't, since 
> I don't believe that's feasible.

> 	- stop sending keepalives after that

I think we're in agreement exept for the interval and the fact that you 
use plural while I'd think a single successful exchange would be 
enough.  :-)

>> Since we don't do anything when the session is active in both 
>> directions or completely idle, reachability tests only happen after a 
>> suspicious active->idle transition. After the test, the multihoming 
>> layer would be satisfied that this transition wasn't because of a 
>> failure, and wouldn't have to do anything until there is a new 
>> active->idle transition.

> not to be a substantial change in the channel.

Wouldn't an outage happening right after a session goes idle and a 
keepalive has been exchanged be such a corner case that we can safely 
ignore it as a special case?

---

From: Joe Touch <[email-masked]>
Date: Wed, 17 Nov 2004 19:50:53 -0800

Iljitsch van Beijnum wrote:
> needs. The multi6 layer needs to know which address pairs work and which 
> don't. There are basically two approaches to this: try every combination 
> and pick the one that loooks best, or start trying combinations until 
> you find the first one that works and use that one. (These are extremes, 
> something in the middle is also possible.) This reachability testing 
> MUST happen in the multi6 layer for robustness and security reasons.
> 
> session is otherwise idle. It doesn't make sense to subcontract this job 
> to the multihoming layer for two reasons:
> 
> multihoming-aware so that it can load balance over more than one address 
> pair at a time in order to achieve x+y throughput rather than max(x,y) 
> throughput, or worse min(x,y)+abs(rand()+0.5)*(max(x,y)-min(x,y)) (ok 
> this means either x or y at random) throughput.

IMO, this is a very bad idea, despite its prevalence in certain 
communities. If this is where you're going, you need TCP to know the 
AGGREGATE of its window, and behave accordingly, but as to 
load-balancing, that's building IP into TCP, and assumes that address 
pairs are meaningful in this context (if there are 8 pairs on each end, 
are there 8 end-to-end pairs, or 64? - there could be 64 unique routes, 
depending on what IP does).

If you want to make a set of links look like one link, build a tunnel 
with a loadbalancing demux in it, but don't go to TCP for that 
functionality, IMO. We looked into it several times before (webmux, in 
particular), and it ends up creating substantial problems with priority 
inversion, layered multiplexing, etc.

Joe

---

From: Iljitsch van Beijnum <[email-masked]>
Date: Thu, 18 Nov 2004 17:56:26 +0100

On 18-nov-04, at 4:50, Joe Touch wrote:

>>> How do we classify when the

>> This is an unsolved problem. I think at some point we need an 
>> interface for this, so that multi6-aware applications can influence 
>> certain aspects of multihoming operation.

I'm assuming you're not disagreeing with this part?

>> I've always assumed that once basic multihoming is in the can the 
>> next step would be to make TCP multihoming-aware so that it can load 
>> balance over more than one address pair at a time in order to achieve 
>> x+y throughput rather than max(x,y) throughput, or worse

> pairs are meaningful in this context

I don't see the problem...?

> (if there are 8 pairs on each end, are there 8 end-to-end pairs, or 
> 64? - there could be 64 unique routes, depending on what IP does).

Obviously a pair can't be on one end. If one host has 4 addresses and 
the other 3, then there are 12 pairs. Communication can happen over one 
pair in both directions (12 possibilities) or different pairs in 
different directions (132 possibilities) for a total of 144 different 
ways to communicate between these hosts.

> functionality, IMO.

So you propose an additional layer of functionality that massages the 
different properties of the different links such that TCP's assumptions 
about the link it runs over (reasonably stable RTT, packet loss = 
congestion) work? This is exactly how we always get into trouble: add 
protocols that break the assumptions of previous protocols. It would be 
much better to change TCP so it has all the information and can make 
the right decisions. Basically this would be like running several TCP 
sessions in parallel, except that we have to ensure in-order delivery. 
Unless this aspect turns out unexpectedly difficult, I see no problems.

> We looked into it several times before (webmux, in particular), and it 
> ends up creating substantial problems with priority inversion, layered 
> multiplexing, etc.

Do you have links to stuff I can look at in this area?

---

From: Joe Touch <[email-masked]>
Date: Thu, 18 Nov 2004 11:31:06 -0800

Iljitsch van Beijnum wrote:
| On 14-nov-04, at 4:29, Joe Touch wrote:
|
|>> But how do you know a session is completely idle vs there is still a
|>> delayed ack coming? (Without looking too deep into TCP, that is.)
|
|> Delayed ACKs come within 200ms of the other end going idle, AFAIR.
|
| Ok. But then we have to account for normal packet loss and normal RTT
| variations.

Agreed,

|>> (Note that is makes little difference whether you use persistent
|>> connections: we work at the IP layer.)
|
|
|> Persistent connections are the kind of TCP connections that might go
|> idle for extended periods;
|
|
| Actually I believe they are closed after a relatively short time when
| they aren't used. And very few sessions use persistent connections last
| time I checked.

TCP connections are not closed just because they are idle, and idle
means no packets exchanged at all.

|> most other TCP connections tend to either be sending data or be closed
|> quickly.
|
|
| A better example of sessions that remain open long even if they're idle
| are SSH sessions.
|
|>> The trick is detecting legitimate idleness because in that state no
|>> action is necessary. For TCP this shouldn't be too hard, for other
|>> protocols we have to find good timeout values. I think these need to
|>> be higher than just a few RTTs.
|
|
|> Such as? I was thinking 300ms-500ms as a ballpark.
|
|
| I was thinking along the lines of 10 - 15 seconds for non-TCP stuff. I
| heaven't really looked in detail, but I'm pretty sure transports for AV
| streaming don't generate ACKs every 500 ms.

Anything that doesn't generate ACKs every couple-hundred ms when the
stream is active is asking for trouble ;-)

| Alternatively, this could be made to auto-tune.

Sure.

| Note though that if you select value X for this and ACKs are sent every
| 1.1*X interval, you'll be sending test packets every X. With a larger X
| this is less likely to happen, but also results in less unnecessary
| traffic if it does.
|
|>> Hm, my assumption was that after a successful test, there wouldn't be
|>> any more tests until something changes.
|
|
|> That presumes async notification from the lower layer. I wasn't, since
|> I don't believe that's feasible.
|
|
|> I.e., I would expect:
|>     - seeing packets exchanged is an in-band test that can be
|>       observed and serves as a keep-alive itself
|>     - send keepalives for the first 500ms after idle
|>     - stop sending keepalives after that
|
|
| I think we're in agreement exept for the interval and the fact that you
| use plural while I'd think a single successful exchange would be
| enough.  :-)

Yes.

|>> Since we don't do anything when the session is active in both
|>> directions or completely idle, reachability tests only happen after a
|>> suspicious active->idle transition. After the test, the multihoming
|>> layer would be satisfied that this transition wasn't because of a
|>> failure, and wouldn't have to do anything until there is a new
|>> active->idle transition.
|
|
|> I'd like it to do something - to keep verifying that reachability is
|> still possible for exactly the period that an idle TCP 'expects' there
|> not to be a substantial change in the channel.
|
| Wouldn't an outage happening right after a session goes idle and a
| keepalive has been exchanged be such a corner case that we can safely
| ignore it as a special case?

I don't like ignoring corner cases, esp. where TCP is concerned.

Joe

---

From: Joe Touch <[email-masked]>
Date: Thu, 18 Nov 2004 11:49:14 -0800

Iljitsch van Beijnum wrote:
| On 18-nov-04, at 4:50, Joe Touch wrote:
|
|>>> How do we classify when the
|>>> connectivity is too poor to be useful - the application should be able
|>>> to determine this and ask for using a different address pair.
|
|
|>> This is an unsolved problem. I think at some point we need an
|>> interface for this, so that multi6-aware applications can influence
|>> certain aspects of multihoming operation.
|
| I'm assuming you're not disagreeing with this part?

Yes.

|>> I've always assumed that once basic multihoming is in the can the
|>> next step would be to make TCP multihoming-aware so that it can load
|>> balance over more than one address pair at a time in order to achieve
|>> x+y throughput rather than max(x,y) throughput, or worse
|
|
|> IMO, this is a very bad idea, despite its prevalence in certain
|> communities. If this is where you're going, you need TCP to know the
|> AGGREGATE of its window, and behave accordingly, but as to
|> load-balancing, that's building IP into TCP, and assumes that address
|> pairs are meaningful in this context
|
| I don't see the problem...?
|
|> (if there are 8 pairs on each end, are there 8 end-to-end pairs, or
|> 64? - there could be 64 unique routes, depending on what IP does).
|
| Obviously a pair can't be on one end.

(sorry - I meant 'addresses on each end')

| If one host has 4 addresses and
| the other 3, then there are 12 pairs. Communication can happen over one
| pair in both directions (12 possibilities) or different pairs in
| different directions (132 possibilities) for a total of 144 different
| ways to communicate between these hosts.
|
|> If you want to make a set of links look like one link, build a tunnel
|> with a loadbalancing demux in it, but don't go to TCP for that
|> functionality, IMO.
|
|
| So you propose an additional layer of functionality that massages the
| different properties of the different links such that TCP's assumptions
| about the link it runs over (reasonably stable RTT, packet loss =
| congestion) work?

No; I'm asserting that the additional functionality is going to be
needed, and we shouldn't step into that mess, since it has already been
stepped in before in the IETF.

| This is exactly how we always get into trouble: add
| protocols that break the assumptions of previous protocols. It would be
| much better to change TCP so it has all the information and can make the
| right decisions.

Er, well, that would argue that TCP should do the multihoming too, not a
shim layer. We already discussed why _that_ isn't the case. For similar
reasons, this is true here too. TCP is _NOT_ the muxing layer. IP is.

| Basically this would be like running several TCP
| sessions in parallel, except that we have to ensure in-order delivery.
| Unless this aspect turns out unexpectedly difficult, I see no problems.
|
|> We looked into it several times before (webmux, in particular), and it
|> ends up creating substantial problems with priority inversion, layered
|> multiplexing, etc.
|
| Do you have links to stuff I can look at in this area?

Check the IETF proceedings for webmux, HTTP-NG, and SOAP.

Joe

---

From: Brian E Carpenter <[email-masked]>
Date: Fri, 19 Nov 2004 10:27:50 +0100

I'm fairly close to declaring this conversation out of
scope. Maybe you can come back to the list when you have
reached a mutual consensus?

    Brian
    with co-chair hat

Joe Touch wrote:
> -----BEGIN PGP SIGNED MESSAGE-----
> Hash: SHA1
> 
> 
> 
> -----BEGIN PGP SIGNATURE-----
> Version: GnuPG v1.2.4 (MingW32)
> Comment: Using GnuPG with Thunderbird - http://enigmail.mozdev.org
> 
> iD8DBQFBnPy6E5f5cImnZrsRAtDpAKCGrKGKCZL9DnsctUGjt9cr0y92KwCgto/G
> Tgg4DhTIbsDF3FwMBdstEXI=
> =hJMa
> -----END PGP SIGNATURE-----
>

---

From: Iljitsch van Beijnum <[email-masked]>
Date: Fri, 19 Nov 2004 11:41:52 +0100

> means no packets exchanged at all.

I was talking about HTTP connections, not about TCP in general.

> stream is active is asking for trouble ;-)

Any particular reason why? I think so many ACKs are excessive.

> I don't like ignoring corner cases, esp. where TCP is concerned.

There is ignoring and then there is ignoring. Obviously when such a 
corner case happens our mechanisms must be able to deal with it in a 
reasonably way. What I'm saying is that not having code to specifically 
handle this case and thus potentially incurring a retransmit before the 
multihoming layer does its thing is ok here.

Don't forget that outages are relatively rare. I don't find the 
situation where there is a significant hiccup when one happens 
problematic, as long as such a hiccup is short enough for sessions to 
recover and for users to wait for this. (I.e., recovering after 239 
wouldn't do, but after 5 - 10 seconds would be just fine.)

---

From: Iljitsch van Beijnum <[email-masked]>
Date: Fri, 19 Nov 2004 11:52:06 +0100

> reasons, this is true here too.

I disagree that these cases are similar enough that the tradeoffs for 
one can be applied to the other.

> | TCP is _NOT_ the muxing layer. IP is.

If memory serves me, port numbers happen in TCP and UDP, not IP... 
(Which is a mistake anyway, IMNSHO.)

But anyway, we have a lot of other bridges to jump off before we reach 
this one, so we can hash this out at some point in the future in the 
then appropriate forum.

> Check the IETF proceedings for webmux, HTTP-NG, and SOAP.

I will, thanks.