Subject: RE: multihoming issues via SCTP

From: Coene Lode <[email-masked]>
Date: Tue, 11 Sep 2001 13:43:49 +0200

a few commments below.

> -----Original Message-----
> From: Iljitsch van Beijnum [mailto:[email-masked]]
> Sent: dinsdag 11 september 2001 11:53
> To: Coene Lode
> Cc: [email-masked]'; [email-masked]'
> Subject: RE: multihoming issues via SCTP
> 
> 
> On Tue, 11 Sep 2001, Coene Lode wrote:
> 
> > > This means there have to be more paths:
> 
> > > A1->X->Y->B1
> > > A1->X->B2
> > > A2->Y->B1
> > > A2->Y->X->B2
> 
> > yes,  but does routing allow them to be used? We got the 
> impression that at
> > a certain time you can have only one path through the 
> network for a certain
> > destination address.
> 
> If we define a path as being from a source interface to a destination
> interface, then the number of paths is only limited by the 
> product of the
> number of interfaces on both sides. Obviously, many of those 
> paths will
> share a lot of infrastructure, but they won't be identical.
> 
> I think the main problem is that in most cases a default route will be
> used, which is tied to one interface. That means the paths 
> originating at
> the different interfaces are effectively identical.

Hmm, not exactly the effect we are looking for.
> 
> > > However, this assumes both that the output interface stays
> > > the same over
> > > the life time of a connection and that the end host is the
> > > one making the
> > > decision how packets are routed to the outside. Both are
> > > unlikely.
> 
> > Well, but that is how it is done with SCTP. SCTP specifies 
> the destination
> > address  and IP should try to route based on that IP address to B.
> 
> Right. So the decision about the output path is done in IP and SCTP
> probably can't control this.
> 
> > If B1 is taken as destination address then the source 
> address should be the
> > corresponding A1(and that is something done by IP in host A).
> 
> I suppose it is possible to do this, although TCP doesn't.

It is not TCP nor SCTP that have to do selection of the source address, They
cannot do this, it is the IP layer who has to do this.
> 
> > > If an
> > > output interface fails, hosts may simply reroute outgoing
> > > packets over the
> > > other interface.
> 
> > That is a possibility, but that depends on who finds it 
> first. If SCTP
> > detects that the interface has gone(via the heartbeats that aren't
> > acknowledge), then it will changeover to another 
> destination address(thus
> > taking the other interface)
> 
> The fact that host B is suddenly sending packets to interface 
> 2 instead of
> 1 of host A, doesn't influence outgoing packets on host A. 
> Assuming B1 and
> B2 are both routed over the same interface (which is likely unless the
> host has a full routing table, something that never happens 
> today) host A
> will have to detect the failure of interface 1 and reroute 
> over interface
> 2.
> 
> > The question for me is: Does it happen that the host 
> changes it routing
> > tables?
> 
> Depends on the host. I think most will do this, but it never hurts to
> experiment. If this doesn't happen automatically when using 
> defaults, it
> is always possible to run RIP.

Well, something to test during a SCTP bakeoff maybe.

> 
> > > And in most cases, routers will reroute packets over
> > > another path if a connection to an ISP fails. So the other
> > > side will often
> > > still see incoming traffic from a remote address that is no longer
> > > reachable for outgoing packets.
> 
> > Uhh, doesn't that mean that then the alternate address 
> should be used
> > instead of the original addresses?
> 
> If the router does this, how would SCTP know a different 
> source address
> has to be used?

It never knows, SCTP only supplies the destination address, the network
layer has to fill in the source address.
> 
> > And as far as I understand something from routing and 
> routing protocols,
> > doesn't that take time?
> 
> Of course. Depends on the interface, though: some can detect 
> failures very
> fast (less than a second) while others won't see failures at 
> all and there
> has to be a timeout in the routing protocol. This takes 30 to 
> 180 seconds.
> 
> The former is likely in point-to-point links, the latter in switched
> environments (Ethernet, ATM).
> 
> > Changing routes in IP networks
> > seem to involve first searching for them and waiting for it 
> till the routing
> > protocol converges. And it is during that convergence faze 
> that the packets
> > will be routed to probably the middle of nowhere(or better 
> said: they are
> > dropped)
> 
> There are many different routing protocols, all with 
> different properties.
> For instance, RIP waits for minutes to decide that a destination is
> unreachable and then removes it from the routing table. OSPF 
> and IS-IS on
> the other hand, will detect all failures in 30 seconds, and interfaces
> that go down immediately. Then the update is flooded through 
> the network
> and all routers recompute the topology. In BGP the detection 
> process is
> much the same but each router recomputes the routing changes 
> individually.
> So in BGP it may take a few moments for routing to stabilize.
> 
> Routers do not necessarily drop packets while updating routing
> information, but they may route them to unreachable destinations or in
> loops for a while.
> 
> > > Also, I don't like the idea of a protocol sending keepalives
> > > over a path
> > > it is otherwise not using. This is a waste of resources and
> > > can trigger
> > > inefficient use of different kinds of caches.
> 
> > Well the alternate routes are not only used for heartbeats, 
> but if the SACK
> > don't get back, then the alternative routes are used for 
> the SACK, so they
> > are not really idle in the strictest sense of the word.
> 
> Well if everything is working, then the other paths aren't 
> used, right?

Yes, that is right

> 
> > What about caches: well caches (in my very humble opinion) 
> shouldn't be
> > there in the first place,
> 
> Some routers use a "route cache" where the most recently used routing
> information is stored. ARP or neighbor discovery are also a form of
> cacheing. Why should processing all of this be triggered just 
> to see if a
> path is available if you don't intend to use it?

You can turn it off(see also Michel's mail)
> 
> > > Also, it won't do much good
> > > unless there are three or more paths of which at least two fail, a
> > > situation that is not likely to be very common.
> 
> > Murphy law: Failures of links, interfaces and routers will 
> happen.(once upon
> > a time, we believed this tale that failures wouldn't 
> happen, but then
> > reality and a lot of users thought us otherwise)
> 
> I'm not saying failures won't happen, just that in most cases the
> heartbeats won't help you. If there are only two paths, 
> you'll have to use
> the second one when the first one fails. So why bother to see if the
> second path is alive? You'll find out soon enough. 

In my view, if the second path is not alive then everything is allready lost
and the net result is that I gain a lot of very, very angry customers.
That is my view and , agreed, for some/many/most/all?(make your choice)
applications that need not to be the case.

> If you 
> have three paths
> and one fails, then each of the others still work so no need to use
> heartbeats again. Only when two paths fail at the same time, 
> it helps to
> know which secondary path has also failed so you know to 
> avoid that one
> and pick the other.
> 
> If doing something and doing nothing yield the same results, 
> the latter is
> preferable.

Depends on the view taken. If you are happy which the situation as you
described then OK, then it is better to do nothing(that is why the heartbeat
is optional). But you will find people(like me) doing heartbeats(and yes,
there are not many of us)

> Iljitsch

Seems like we are in between a rock and a hard place. Maybe I'll try to come
up with something.
Thanks anyway for the comments. They cleared some things for me. 

Yours sincerly,
Lode

---

From: Iljitsch van Beijnum <[email-masked]>
Date: Wed, 12 Sep 2001 17:53:49 +0200 (CEST)

> Hmm, not exactly the effect we are looking for.

I believe SCTP was originally conceived as a protocol to transport
telephony signalling. In such a context, it is possible to create a
network where this is not a problem. However, I gather that there are
people who want to do a lot more with SCTP and the Internet as a whole is
not likely to be reengineerd to eliminate this problem.

> > > If B1 is taken as destination address then the source  address should be the

> cannot do this, it is the IP layer who has to do this.

Please keep in mind that selection of the source address and of the source
interface are not related. In TCP, the system will select the source
address matching the output interface for the first packet of an outgoing
session. Since the remote TCP expects this address to stay the same,
subsequent packets within a session all have the same source address,
regardless of the output interface.

Since it is possible in TCP to find a source address based on the output
interface for the first packet, it stands to reason that SCTP could do
this for every packet.

[heatbeats]

> You can turn it off(see also Michel's mail)

I suggest that for implementations that are deployed in regular hosts that
are expected to communicate over the Internet, heartbeats are disabled by
default.

> applications that need not to be the case.

I don't understand.

Either the line is down or it is up. Heartbeating will not change this:
we're not talking about quantum physics, where looking at something
changes its state.

I suppose that in telephony signalling, where there the number of sessions
is very small and they stay up for very long and the data is very
important, it can make sense to immediately see when a path goes down. But
now consider a web server. It has dozens of simultaneous sessions, and
each only lives for a few moments. Is it then sensible to send heartbeats
over dozens of alternative paths? I hardly think so: having each session
discover the same situation individually doesn't make any sense. This
should be done at a lower level, if this functionality is required. But I
that goes for all the multihoming features of SCTP, IMHO.

> > the latter is preferable.

> there are not many of us)

Please read what I'm saying. If not asking for anything will get yo a cup
of tea. And if asking for a cup of coffee will get you a cup of tea also.
Is it then worth the trouble asking for coffee?

> Seems like we are in between a rock and a hard place. Maybe I'll try to come

Not really. Have a look at http://www.muada.com/sctp.gif

[1st drawing] It describes a situation where host A communicates with host
B and both networks have to routers and connect to two ISPs. As you can
see, the default makes all the output traffic go over one connection:

Red:	A1 -> ISP W -> ISP Y -> B3
Orange:	B3 -> ISP Y -> ISP W -> A1
Blue:	A1 -> ISP W -> ISP Z -> B4
Cyan:	B3 -> ISP Y -> ISP X -> A2

[2nd drawing] Then, if the 1 interface on A fails, it will detect this
(hopefully) and point the default route to the other router over the other
interface. Now the orange path from B to A fails, but the red and blue
paths from A to B both still work and the cyan path from B to A as well.

[3rd drawing] Another type of failure would be where ISP W and ISP Y (if
they are the same ISP or share something) both fail. Then, two paths fail
and the remaining paths get somewhat longer than necessary, but there is
still a path in each direction.

[4th drawing] In an environment where both ends of the connections can be
controlled or coordinated, it is very easy to add static routes from A to
B over A1/B3 and A2/B4 so the paths are more diverse and have to depend
less on reliability/redundancy measures elsewhere in the network. If the
network interfaces of A and B are in the same subnet, this even happens
automatically without static routes.

Iljitsch van Beijnum