Subject: [arch-d] 2. thinking about an implementation

From: Iljitsch van Beijnum <[email-masked]>
Date: Fri, 12 Jan 2007 18:43:26 +0100

Since none of us have the power to transform the internet over night,  
simply implementing the architecture that I outlined in my previous  
message, or any architecture that isn't 98% an enumeration of the  
current state of the network, isn't possible. So it's necessary to  
take that current state into consideration, and, more importantly,  
come up with solutions for today's problems, which I'll take to be  
the problems identified by the IAB routing and addressing workshop.

The problem that we have is that the core routers need to forward  
packets very fast and they need to consider a lot of state doing it.  
This makes future growth very expensive. Either reducing the number  
of packets / forwarding speed or reducing the amount of state will  
keep the internet affordable. Alternatively, letting the people that  
benefit from the combined speed/state explosion pay makes the issue  
an economic one that will be solved through garden variety  
capitalism. We already have a reasonable model for paying for  
bandwidth, so we would have to come up with a model for paying for  
state, which is mostly multihoming, using provider independent  
address space and bad aggregation. Nobody has been able to suggest  
such a model that fits with the current structure of the network. So  
we're back on the technical side.

Implied in the "jack up" term is the notion that there needs to be a  
completely new level. This is a problem. Introducing something new  
throughout the entire internet is pretty much a non-starter even in  
the long term. What we need is something that can be applied to the  
parts of the network where the pain is felt. This is in the large  
networks that use the fastest routers. Interestingly, for the most  
part, the large networks that have fast routers don't have or need  
fast routers everywhere: there is generally an aggregation / customer  
facing group of routers that is much more modest, and which don't  
need to keep the full BGP default-free state. The problems start when  
all the traffic going to all possible destinations in the world comes  
together. This is where both the speed and the state is required.

There are several ways to avoid this speed/state combination:

1. Parallelization: have several fast routers that each have part of  
the state. So the "slow" routers send traffic for 0.0.0.0/2 to core  
router A, 64.0.0.0/2 to B, 128.0.0.0/2 to C and 192.0.0.0/2 to D.  
Core routers A - D only need a quarter of the full BGP table each.

2. Tunneling: have the "slow" routers encapsulate the traffic (not in  
IP, please, ethernet or MPLS is much more efficient!) and send it  
directly to the border of the network rather than have core routers  
look at it.

3. Internal aggregation: have a border router that handles 0.0.0.0/2,  
one that handles 64.0.0.0/2 and so on. This works much better if  
address allocation is non-random (see http://www.muada.com/drafts/ 
draft-van-beijnum-multi6-isp-int-aggr-01.txt ) but it works with  
current addressing as well if the resulting stretch is considered  
acceptable.

(Observe that moving packets as per an aggregate or as per a tunnel  
address accomplishes the exact same thing.)

4. 90/10 rule: determine the 10% destination prefixes that suck up  
the most traffic (could be 90%) and put those in the fast routers,  
keep the other 90% of the prefixes that are the destination for the  
remaining 10% of the traffic in much slower routers that can use much  
cheaper technology. The jack up/down here is dynamic: remove a prefix  
form a fast router and packets toward it are automatically handled by  
the slow routers.

Note that all of these are internal to an autonomous system and  
therefore deployable. More gains can be had by having ASes cooperate  
by having the source AS encapsulate and have the destination AS  
decapsulate. For this, we need new routing/signaling which we can  
optimize specifically for this task so it should be able to handle  
considerably more prefixes without trouble than what we can currently  
do today with BGP. And since the inter-AS encapsulation is an  
additional optimization rather than a required first step, this too  
should be reasonably deployable.

The third step is allowing end-users to participate in the  
encapsulation/decapsulation which should allow adding additional  
features such as mobility and end-user TE/optimization. And suddenly  
we're close to the "negotiate information at every step" model, too.

_______________________________________________
Architecture-discuss mailing list
[email-masked]
https://www1.ietf.org/mailman/listinfo/architecture-discuss

---

From: John Leslie <[email-masked]>
Date: Fri, 12 Jan 2007 16:23:09 -0500

> packets very fast and they need to consider a lot of state doing it.  

   I agree 100%.

> keep the internet affordable.

   Well said!

> Alternatively, letting the people that benefit from the combined
> speed/state explosion pay makes the issue an economic one that will
> be solved through garden variety capitalism.

   Alas, it's hard to agree who should be asked to pay...

> We already have a reasonable model for paying for bandwidth, so we
> would have to come up with a model for paying for  state,

   Good point.

> which is mostly multihoming, using provider independent address space
> and bad aggregation.

   Well, No... "State" is stuff which changes. Especially, "state" is
stuff which changes fast enough that synchronizing it is non-trivial.

> Nobody has been able to suggest such a model that fits with the
> current structure of the network. So we're back on the technical
> side.

   No. We're into an area where architecture can really help.

> the long term.

   Actually, the problem with "jack-up" is that it _increases_ the
"state" information we're likely to need to synchronize. What's a
non-starter is designing around a "flag day".

> What we need is something that can be applied to the parts of the
> network where the pain is felt. This is in the large networks that
> use the fastest routers.

   That's by no means the only place the pain is felt...

> Interestingly, for the most part, the large networks that have fast
> routers don't have or need fast routers everywhere: there is generally
> an aggregation / customer facing group of routers that is much more
> modest, and which don't need to keep the full BGP default-free state.

   That seems obvious to us. Alas, it's not as obvious to the management
of the companies operating those routers. Furthermore, BGP thrashing
inflicts pain much more widely than that paradigm would suggest...

> The problems start when all the traffic going to all possible
> destinations in the world comes together.

   (Which, of course, need not happen in the first place -- as you
later observe...)

> This is where both the speed and the state is required.

   This is where speed is required. "State" could actually be relaxed
a bit here, if we could escape telco-think and agree that state changes
_will_ interrupt forwarding...

> 1. Parallelization...
> 2. Tunneling...
> 3. Internal aggregation...
> 4. 90/10 rule...

   These and other tricks can be made to handle the speed issue --
which is why I have said that mere size of the forwarding table isn't
enough to justify this effort. It's synchronizing the "state" that
can't scale.

> should be reasonably deployable.

   Indeed. (Though this requires _some_ architectural work first...)

> features such as mobility and end-user TE/optimization.

   This is where I lose you. I do believe there are things we could
handle end-to-end, but encapsulation & aggregation aren't among them.

--
John Leslie <[email-masked]>

---

From: Iljitsch van Beijnum <[email-masked]>
Date: Sat, 13 Jan 2007 00:18:31 +0100

>    Alas, it's hard to agree who should be asked to pay...

Actually the really hard part would be to determine who gets the  
money... This only works when the actual extra work that comes from  
using PI or multihoming must be done by the PI user / multihomer, I  
don't think pricing is a usable mechanism here. But of course I'm not  
an economist, nor do I play one on TV.

> stuff which changes fast enough that synchronizing it is non-trivial.

There are two parts: the synchronizing and then the lookups at packet  
forwarding time. Both have the potential to be problematic, although  
I think the former is more easily parallelizeable and better  
protocols will also help a lot. With the BGP hammer every possible  
type of event looks like a routing update nail and the nails have the  
ugly tendency to multiply to boot.

> "state" information we're likely to need to synchronize.

Volume in itself isn't all that relevant. 100 spam emails a day is an  
annoyance, 100 spam phone calls a day is a full time job.  
Disseminating a lot of state information in and of itself isn't  
immediately fatal if we can avoid having to update it very often or  
having to do lookups on it very fast. I.e., if an edge box that lives  
at 1 Gbps or lower bandwidth only has to encapsulate packets without  
considering reachability because rerouting at the outer header  
protocol takes care of that, it could work well.

> What's a non-starter is designing around a "flag day".

It's even worse. Going from IPv4 to IPv6 with a flag day won't work.  
But going from IPv4 to dual stack and then from dual stack to IPv6  
doesn't work either, even though no flag day is required, because in  
a network the size of the internet it's impossible to complete any  
action that involves changes to every { host | stack | router | AS  
| ... } . There will always be holdouts.

> handle end-to-end, but encapsulation & aggregation aren't among them.

The first step is to have edge (customer-facing) routers in ISP  
networks encapsulate and border routers decapsulate in order to make  
packets flow through the core without the core needing to have the  
90% prefixes that generate 10% of the traffic.

The second step is to encapsulate at the source edge and decapsulate  
at the destination edge, so the border routers don't have to see the  
"vanity prefixes" either.

The third step is to optionally have customers encapsulate and/or  
decapsulate. This saves capacity on the ISP edge routers and allows  
the customer equipment to make smarter decisions.

Compare this to the likes of shim6 where the hosts perform the  
encapsulation/decapsulation so the whole thing is transparent to ALL  
routers.

The big advantage of doing all of this in ISP networks is that that  
way, customers don't get to see their locator addresses so they  
aren't tempted to hardcode those all over the place. With something  
like shim6 you still have the renumbering problem which creates the  
need for PI which creates the routing state explosion.

---

From: John Leslie <[email-masked]>
Date: Sat, 13 Jan 2007 17:54:11 -0500

Iljitsch van Beijnum <[email-masked]> wrote:
>> ... "State" is stuff which changes. Especially, "state" is stuff
>> which changes fast enough that synchronizing it is non-trivial.
> 
> forwarding time.

   If the lookup isn't local, we (probably) haven't synchronized.

> Both have the potential to be problematic, although I think the former
> is more easily parallelizeable and better protocols will also help a
> lot.

   Perfect synchronization isn't possible throughout the routing fabric.

> With the BGP hammer every possible type of event looks like a routing
> update nail and the nails have the ugly tendency to multiply to boot.

   Well, yes, we could do better than BGP... But architecture is about
avoiding any need to use BGP-like tools where they can't work well.

> ... Disseminating a lot of state information in and of itself isn't  

   Indeed, edge routers could afford to synchronize more state than
backbone routers can.

> | ... } . There will always be holdouts.

   Going from IPv4 to dual-stack _does_ work. (I agree IPv4 won't
disappear anytime soon.)

> 90% prefixes that generate 10% of the traffic.

   This is a rather big step; and it doesn't feel necessary...

> "vanity prefixes" either.

   What do you mean by "edge"? (This could be a critical architectural
question.)

> the customer equipment to make smarter decisions.

   Designing the architecture for this looks like an interesting
problem.

---

From: Bruce Curtis <[email-masked]>
Date: Tue, 16 Jan 2007 18:38:50 -0600

>    Indeed, edge routers could afford to synchronize more state than

   I'm not sure I agree that edge routers could afford to synchronize  
more state than backbone routers can.  A lot of edge routers might  
have to be beefed up to contain the state for the whole global  
routing table and keep up with the churn.

   Also if churn is really the problem in the core, then moving the  
churn to the edge doesn't necessarily solve the problem.  It may just  
be moving and delaying the problem, only providing short-term relief.

---
Bruce Curtis                         [email-masked]
Certified NetAnalyst II                701-231-8527
North Dakota State University

---

From: John Leslie <[email-masked]>
Date: Wed, 17 Jan 2007 13:48:43 -0500

> routing table and keep up with the churn.

   We're talking architecture: it might indeed be necessary to beef up
edge routers. (That's somebody else's problem.)

   But it shouldn't be necessary for them to hold the "whole global
routing table", most of which is of no interest to them. Even if this
were necessary, _holding_ a global routing table several orders of
magnitude larger than today's "defaultless" tables isn't difficult.

   It is clearly _not_ necessary for them to keep up with the churn:
they're "edge routers" which don't have to pass routes to neighbors.

> be moving and delaying the problem, only providing short-term relief.

   I don't believe either Iljitsch or I have proposed "moving churn to
the edge". Nonetheless "churn at the edge" is a much more scalable
issue, IMHO.

---

From: Iljitsch van Beijnum <[email-masked]>
Date: Thu, 18 Jan 2007 12:55:30 +0100

>    Going from IPv4 to dual-stack _does_ work. (I agree IPv4 won't

I'm not sure what you mean by "does work". Certainly, it's possible  
to upgrade stuff from IPv4 to dual stack. But waiting for the whole  
IPv4 internet to become dual stack before we get to turn off IPv4  
isn't a realistic IPv6 deployment scenario.

>    This is a rather big step; and it doesn't feel necessary...

Seems like a relatively small step to me...

>    What do you mean by "edge"? (This could be a critical architectural

In general the place where end-user and ISP infrastructures come  
together, in this particular case the ISP side of it.

>    Designing the architecture for this looks like an interesting

A lack of interesting problems to work on would be a bad thing.  :-)

---

From: John Leslie <[email-masked]>
Date: Thu, 18 Jan 2007 15:19:19 -0500

> isn't a realistic IPv6 deployment scenario.

   Personally I agree. (It might be appropriate to discuss more practical
deployment options...)

> Seems like a relatively small step to me...

   Anything that requires coordination of both ends is big, IMHO.

> together, in this particular case the ISP side of it.

   At first blush, this seems a poor choice. There's more benefit to
the user than the ISP; and interoperability among mass-market products
is likely to be easier to attain than interoperability among one-off
kludges by different ISPs.

---

From: Bruce Curtis <[email-masked]>
Date: Sat, 20 Jan 2007 04:54:21 -0600

> edge routers. (That's somebody else's problem.)

    If there are problems with scaling in the core today then  
solutions that increase state in edge routers might not scale in the  
future.  Architectures that push as much state as possible all the  
way to the endpoints might scale the best.

> magnitude larger than today's "defaultless" tables isn't difficult.

   I might not be following correctly but a lot of proposals have  
depended on choosing  one of several possible destination locators at  
the edge routers.  If the edge routers don't have info about the  
state of all the links to all multihomed sites it might send packets  
to a locator that was only reachable on a link that was down.

   The report on the IAB Workshop on Routing and Addressing mentions  
that all locator/identifier split proposals must address the problem  
of detecting locator failures and redirecting data flows to remaining  
locators for a multihomed site.

> the edge".

   Sorry, I didn't mean to imply that anyone proposed it exactly that  
way.  That is just a danger that I see in architectures that move the  
problem of handling "locator failures" from the core routers to the  
edge routers.  I understand how moving to a split locator/identifier  
and hierarchical routing reduces the size of the routing table but I  
don't see how it reduces the amount of information about paths to  
multihomed sites that are down that must be disseminated to either  
the core routers or the edge routers.

> Nonetheless "churn at the edge" is a much more scalable
> issue, IMHO.
>
> --

---

---

From: John Leslie <[email-masked]>
Date: Sat, 20 Jan 2007 14:25:18 -0500

Bruce Curtis <[email-masked]> wrote:
> If there are problems with scaling in the core today then solutions
> that increase state in edge routers might not scale in the future.
> Architectures that push as much state as possible all the way to
> the endpoints might scale the best.

   Yes.

> to a locator that was only reachable on a link that was down.

   I hope we're not seriously conlidering any requirement that a packet
in transit deserves to be re-routed if we find its next-hop down.

   (In fact, I claim the distinction between "congested" and "down" is
arbitrary, and that such a distinction should be made by end systems.)

> locators for a multihomed site.

   My view on this is that end systems should evaluate congestion on
all options, and migrate traffic to the least congested choice.

   (YMMV, I suppose...)

> the core routers or the edge routers.

   I'd prefer to move it to end systems, actually. Edge routers _might_
be part of a transition.

---

From: Bruce Curtis <[email-masked]>
Date: Sat, 20 Jan 2007 16:16:39 -0600

> arbitrary, and that such a distinction should be made by end systems.)

   I'm not sure about others but I have thought about such options.   
It helped me think through the issues and differences.  But the more  
I think about it the more it seems that perhaps the endpoint is the  
better place for such decisions, endpoints today already have to deal  
with one of multiple addresses associated with a DNS name being down  
when sessions are initiated.

   If we put all of the destination locators in a packet then core  
routers have enough information to re-route if a next-hop is down  
without requiring state and churn in either the core or edge  
routers.  But there are a few complications required in such a  
solution such as the core routers rewriting the list of destination  
locators or at least a bit to mark some as failed etc.

   It still might be a good idea to consider putting all of the  
destination locators in a packet for Traffic Engineering but if the  
end points are responsible for locator failure things are a lot  
simpler in the core and in the edge.

> all options, and migrate traffic to the least congested choice.

   I think that it would be an advantage for an architecture to have  
that option.  But to satisfy everyone I think the architecture should  
also allow the option of traffic engineering at the edge and in the  
core.

>    I'd prefer to move it to end systems, actually. Edge routers  

  Sounds like the possibility of handling locator failure in the end  
system is part of the discussion.

  Thanks.

---

---

From: Brian E Carpenter <[email-masked]>
Date: Sun, 21 Jan 2007 09:25:11 +0100

>> the endpoints might scale the best.

Please note, though, that one of the more interesting criticisms
of shim6 is that it asks end systems (e.g. server farms) to
hold extra state per session. Moving state around does move
heat dissipation around.

> arbitrary, and that such a distinction should be made by end systems.)

That sounds plausible, but the real problem is distinguishing "congested"
from "lossy", because that dramatically affects how a transport
protocol should react - slow down in the case of "congested", rapid
retransmit in the case of "lossy".

     Brian

---

From: "Spencer Dawkins" <[email-masked]>
Date: Sun, 21 Jan 2007 07:18:55 -0600

OK, we've been here before, too.

> retransmit in the case of "lossy".

We have about thirty years of experience (which I missed the first tweny 
years of, but which people then spent ten years reminding me about) with 
trying to distinguish between congestion and corruption, and responding 
accordingly.

We made this harder for ourselves than it had to be with TCP (one indication 
that there's a problem - packet loss - and two problems that can cause that 
indication - congestion, and corruption).

Adding explicit congestion notification (ECN) helps (potentially), but not 
as much as we might hope - we don't know how to convey ECN at very high 
congestion levels, so an endpoint can tell that there is congestion if ECN 
IS set, but can't interpret the lack of ECN-marked traffic as an absence of 
congestion, since all the ECN-marked packets could have been discarded due 
to congestion - this would be the precise wrong time to rapidly retransmit.

We have dorked around with explicit error notification, but since bit errors 
may be present in the IP header fields, it's not precisely obvious where to 
send explicit error notifications ("I received an errored packet with your 
IP address as source, but you might not have sent it, because the packet 
didn't pass checksum processing").

We try to be conservative (having previously tried "when you get a timeout, 
feel free to retransmit all outstanding packets" and experiencing the 
resulting meltdown - Dave Clark referred to this as the "kick 'em when 
they're down" school of transport retransmission), and the problem here is 
that if you spend a few RTTs trying to be conservative, you end up 
recovering about as quickly if you assume that everything is congestion as 
you do with any other strategy.

Perhaps some careful thought on transport might be part of a new 
architecture?

We talked about this at the TERNLI ad hoc meeting in Montreal, and Sally and 
Pasi produced draft-sarolahti-tsvwg-crosslayer-00.txt, but I haven't seen 
anything past that...

Spencer

p.s. the crosslayer draft does a better job of characterizing IETF work in 
this space in Section 5 - the above is just my recollection...

---

From: Wesley Eddy <[email-masked]>
Date: Sun, 21 Jan 2007 10:12:48 -0500

On Sun, Jan 21, 2007 at 07:18:55AM -0600, Spencer Dawkins wrote:
> 
> We have dorked around with explicit error notification, but since bit 
> errors may be present in the IP header fields, it's not precisely obvious 
> where to send explicit error notifications ("I received an errored packet 
> with your IP address as source, but you might not have sent it, because the 
> packet didn't pass checksum processing").

To get around this, we've also tried explicit error _rate_ notification,
so we don't have to know exactly who to send an error report for a
particular errored packet to.  Congestion control based on the relative
congestion and corruption rates to the total loss rate seems to work
nicely.  Of course, this requires some router support. 

Two references:

http://www.icir.org/mallman/papers/eten-comnet.ps
Rajesh Krishnan, James Sterbenz, Wesley Eddy, Craig Partridge, Mark
Allman. Explicit Transport Error Notification (ETEN) for Error-Prone
Wireless and Satellite Networks. Computer Networks, 46(3), October 2004. 

http://www.icir.org/mallman/papers/ceten-ccr-oct2004.ps
Wesley Eddy, Shawn Ostermann, Mark Allman. New Techniques for Making
Transport Protocols Robust to Corruption-Based Loss. ACM Computer
Communication Review, 34(5), October 2004. 

-- 
Wesley M. Eddy
Verizon Federal Network Systems

---

From: Henning Schulzrinne <[email-masked]>
Date: Sun, 21 Jan 2007 12:45:41 -0500

I'm not sure how practically relevant corruption loss is outside some  
corner cases:

- Extremely high speed TCP connections, where even bit error rates of  
10e-12 matter and thus normal error detection mechanisms don't work  
well any more and where even very rare losses cause havoc with the  
congestion control algorithm;

- Specialty links, such as deep-space satellites

- 3G radio links when configured for packet audio (but these wouldn't  
be used for TCP-like protocols, one would hope)

Are there modern, commonly-occurring links that don't retransmit  
errored packet locally for data traffic and have significant error  
rates?

Henning

On Jan 21, 2007, at 8:18 AM, Spencer Dawkins wrote:

> We have about thirty years of experience (which I missed the first  
> tweny years of, but which people then spent ten years reminding me  
> about) with trying to distinguish between congestion and  
> corruption, and responding accordingly.
>
> We made this harder for ourselves than it had to be with TCP (one  
> indication that there's a problem - packet loss - and two problems  
> that can cause that indication - congestion, and corruption).
>
> Adding explicit congestion notification (ECN) helps (potentially),  
> but not as much as we might hope - we don't know how to convey ECN  
> at very high congestion levels, so an endpoint can tell that there  
> is congestion if ECN IS set, but can't interpret the lack of ECN- 
> marked traffic as an absence of congestion, since all the ECN- 
> marked packets could have been discarded due to congestion - this  
> would be the precise wrong time to rapidly retransmit.
>
> We have dorked around with explicit error notification, but since  
> bit errors may be present in the IP header fields, it's not  
> precisely obvious where to send explicit error notifications ("I  
> received an errored packet with your IP address as source, but you  
> might not have sent it, because the packet didn't pass checksum  
> processing").
>
> We try to be conservative (having previously tried "when you get a  
> timeout, feel free to retransmit all outstanding packets" and  
> experiencing the resulting meltdown - Dave Clark referred to this  
> as the "kick 'em when they're down" school of transport  
> retransmission), and the problem here is that if you spend a few  
> RTTs trying to be conservative, you end up recovering about as  
> quickly if you assume that everything is congestion as you do with  
> any other strategy.
>
> We talked about this at the TERNLI ad hoc meeting in Montreal, and  
> Sally and Pasi produced draft-sarolahti-tsvwg-crosslayer-00.txt,  
> but I haven't seen anything past that...
>
> Spencer
>
> p.s. the crosslayer draft does a better job of characterizing IETF  
> work in this space in Section 5 - the above is just my recollection...
>
>

---

From: Andrew McGregor <[email-masked]>
Date: Mon, 22 Jan 2007 11:09:38 +1300

Network-encoded mesh wireless systems will be becoming available  
soon, and they do have the property of not doing local retransmit  
(because noone inside the subnet has the whole packet, except the  
source and the edge router).  If things are working badly the error  
rate could be pretty high too, although you'd hope that doesn't  
happen often.

Andrew

On 22/01/2007, at 6:45 AM, Henning Schulzrinne wrote:

> I'm not sure how practically relevant corruption loss is outside  
> some corner cases:
>
> - Extremely high speed TCP connections, where even bit error rates  
> of 10e-12 matter and thus normal error detection mechanisms don't  
> work well any more and where even very rare losses cause havoc with  
> the congestion control algorithm;
>
> - 3G radio links when configured for packet audio (but these  
> wouldn't be used for TCP-like protocols, one would hope)
>

---

From: "Spencer Dawkins" <[email-masked]>
Date: Sun, 21 Jan 2007 22:34:15 -0600

> Network-encoded mesh wireless systems will be becoming available  soon, 
> and they do have the property of not doing local retransmit  (because 
> noone inside the subnet has the whole packet, except the  source and the 
> edge router).  If things are working badly the error  rate could be pretty 
> high too, although you'd hope that doesn't  happen often.
>
> Andrew

Oh, goody! This sounds like a wonderful place to run current reliable 
transport protocols :-)

But, in response to Henning's note...

Another issue in Trigtran (which I can talk about with some confidence) was 
also that once you lose a link and the sender has gone through a few 
wait-twice-as-long-and-retry-again cycles, it also takes a while after a 
link comes back up before the sender figures that out. So this wasn't just a 
throughput question, it also involved how quickly you respond to changes in 
the set of potential paths.

I can't promise that the world won't be so overdeployed with WiMax that 
you'll never be somewhere that you can't send and receive traffic, but so 
far, this has not been my experience :-)

Spencer

>> 10e-12 matter and thus normal error detection mechanisms don't  work well 
>> any more and where even very rare losses cause havoc with  the congestion 
>> control algorithm;
>>
>> - 3G radio links when configured for packet audio (but these  wouldn't be 
>> used for TCP-like protocols, one would hope)
>>
>> Are there modern, commonly-occurring links that don't retransmit  errored 
>> packet locally for data traffic and have significant error  rates?
>>
>> Henning

---

From: Bruce Curtis <[email-masked]>
Date: Mon, 22 Jan 2007 01:55:22 -0600

> hold extra state per session. Moving state around does move

   It would seem to me that if the state is too much for servers in  
server farms to handle then it would likely be too much for the edge  
router for the server farm to handle also.  But I don't run server  
farms so am far from an expert on the topic.

   If the server farm folks think it would be preferable to have the  
state in the edge router or a middle box it would be nice if the  
architecture allowed that option without too much difficulty but  
personally I would prefer to consider the server farms a special case  
and have the default to be to handle the state in the endpoints.

---

---

From: John Leslie <[email-masked]>
Date: Mon, 22 Jan 2007 10:46:25 -0500

Spencer Dawkins <[email-masked]> wrote:
> [Brian Carpenter wrote:]
>> [John Leslie wrote:]
> indication that there's a problem - packet loss - and two problems that
> can cause that indication - congestion, and corruption).

   (Actually, "at least two"...)

> Adding explicit congestion notification (ECN) helps (potentially), but
> not as much as we might hope -

   Alas, I'm not even sure "ECN" helps. :^(

   We've got to _start_ by measuring "congestion", not measuring "packet
loss" and calling it congestion.

   ECN, IMHO, didn't go far enough in that direction. RFC3168 defines
ECN as
" 
" a congestion indication for incipient congestion (as in RED and
" earlier work [RJ90]) where the notification can sometimes be through
" marking packets rather than dropping them.

which, IMHO, implies that it's a last-ditch effort to avoid signaling
"congestion" by dropping a packet.

   I can't bring myself to call that "measuring" congestion.

> we don't know how to convey ECN at very high congestion levels, so an
> endpoint can tell that there is congestion if ECN IS set, but can't
> interpret the lack of ECN-marked traffic as an absence of congestion...

   To be fair, ECN (though misnamed IMHO) tried only to be an additional
tool in our bag-o-tricks to detect congestion. The absence of ECN-marked
packets, clearly, should not be taken to mean anything at all.

> errors may be present in the IP header fields, it's not precisely
> obvious where to send explicit error notifications ("I received an
> errored packet with your IP address as source, but you might not have
> sent it, because the packet didn't pass checksum processing").

   IP contains a computationally cheap error indication (checksum).
It _is_ obvious, actually, where an IP error indication should be
sent: to whatever machine forwarded it to you. (It's not obvious,
OTOH, that there's any point in sending it...)

> experiencing the resulting meltdown...

   ;^((

> the problem here is that if you spend a few RTTs trying to be
> conservative, you end up recovering about as quickly if you assume
> that everything is congestion as you do with any other strategy.

   ... in the absence of actual measurement of congestion, that is...

> Perhaps some careful thought on transport might be part of a new 

   I certainly hope so!

> We talked about this at the TERNLI ad hoc meeting in Montreal, and Sally 
> and Pasi produced draft-sarolahti-tsvwg-crosslayer-00.txt, but I haven't 
> seen anything past that...

   A lot of good summary in that, but the title is unfortunate, giving
the impression that we must look to layering invasions for a solution.

   I found draft-briscoe-tsvwg-re-ecn-tcp particularly interesting...

---

From: Simon Leinen <[email-masked]>
Date: Mon, 22 Jan 2007 18:50:36 +0100

Brian E Carpenter writes:
> Please note, though, that one of the more interesting criticisms of
> shim6 is that it asks end systems (e.g. server farms) to hold extra
> state per session. Moving state around does move heat dissipation
> around.

What amount of additional state are we talking about? I don't know
shim6 well enough to assess this, but my gut feeling says this cannot
be a showstopper.

Servers already store quite a big of per-client (even per-connection)
state, typically in the form of TCP state including retransmission
buffers, which will have to be quite large if clients can be far away
and high per-session throughput is desired.  I would be very surprised
if shim6 support would add significantly to state requirements if TCP
is used.

And even where additional server state for multihoming clients is
prohibitive (maybe for popular DNS servers that mostly use UDP), I
think this should be solvable by servers refusing to store clients'
shim6 state beyond the absolutely necessary.  This would deny seamless
failover, but since such protocols won't have valuable state this
shouldn't hurt too much.

So, I cannot see why something like shim6 shouldn't work with busy
servers.

(I also agree with Bruce Curtis' points.)
-- 
Simon.

---

From: Jari Arkko <[email-masked]>
Date: Mon, 22 Jan 2007 21:15:43 +0200

Hi Simon,
>
> What amount of additional state are we talking about? 
The amount of state seems relatively small -- basically session
identifiers, upper layer identifiers, locators, security information,
and some administrative pieces of information such as generation
counters for the latest list of current locators, retransmission
timers for control messages, etc. Details in draft-ietf-shim6-proto,
Section 6.1. For other host based mechanisms these numbers
may vary quite a bit. HIP, for instance, needs more memory
because it does more, e.g., establishes end-to-end ESP state.
Of course, if you multiply this amount of state information
by a large number of clients, you may need a lot of memory
to hold the state and a lot of signaling to set it up.

In any case, perhaps more important issues are whether
this information is needed for all communications, and
who gets to decide whether the state has to be registered.

Holding state in the DNS server for its clients seems
silly, for instance, if the transactions consist of one
message and a reply. It may make more sense to
send a new message after a re-homing event than
to try to keep the server aware of your situation.
The same may apply to other applications too.

It also seems important that whoever is requested
to spend effort or keep state has a say in whether
they are willing to do so. My understanding is that
this is what is done in all host based mechanisms
to date, i.e., the peer can refuse to keep your
multihoming state.

Jari

---

From: Brian E Carpenter <[email-masked]>
Date: Tue, 23 Jan 2007 14:28:17 +0100

> What amount of additional state are we talking about? 

I like Jari's answer, but I would suggest another way to look
at it: instead of the site NAT (or PAT) having to hold state
per active flow, each host will have to hold state per active
flow. So the border box will get noticeably less hot and (in a
server farm) the offload NICs will each get a little hotter.

    Brian

---

From: Iljitsch van Beijnum <[email-masked]>
Date: Mon, 29 Jan 2007 17:42:15 +0100

> corruption, and responding accordingly.

You are looking at the symptom rather than the disease. The real  
problem is lack of data flow. Both having congestion and trying to  
avoid congestion cause this, and when packet loss is the result of  
errors, trying to avoid congestion doesn't help either.

I think we've reached the point where trying to avoid congestion is  
doing more harm than good. (See all the work on long fat pipes and  
more aggressive TCP ramp up.) It would be so much easier if we could  
just have a "discard eligible" bit and be done with it.

Note by the way that most errors cause only a single lost packet once  
in a while so fast retransmit will catch most of those. If you keep  
losing multiple successive packets because of errors you may want to  
spend time on something other than TCP performance.

> Perhaps some careful thought on transport might be part of a new  

TCP is too stupid. It knows nothing about timing and uses windows to  
make everything work. Maybe if I have a 1 Gbps link to a router with  
a 100 Mbps uplink, it is be a good idea to send packets at 100 Mbps  
rather than burst at 1 Gbps 1/10th of the time?

For a while we had a buffer arms race between routers and TCP, but  
TCP won because the routers got too fast. What I mean is that  
ideally, TCP buffers should be equal to the bandwidth/delay product.  
But they're often much larger, so traffic is burstier than necessary.  
In order to accommodate the bursts, routers have large buffer pools.  
But at 10 Gbps this is an expensive proposition so routers can't  
accommodate TCP's excessive burstiness under all circumstances.