Subject: Re: [RAM] Comment on draft-farinacci-lisp-00.txt (LISP)

From: Tony Li <[email-masked]>
Date: Sun, 1 Apr 2007 00:19:15 -0700

On Mar 31, 2007, at 6:31 PM, Ross Callon wrote:

>  (1) You are going to make the data plane MUCH slower; or
>  (2) You are going to very lightly load the data plane; or
>  (3) You are going to implement the control aspects of LISP
>     somewhere other than on the central processor; or
>  (4) You are going to build routers whose central processor is
>     several orders of magnitude faster than on current routers; or
>  (5) You feel that you can implement LISP on a PE router
>     with a load on the control plane which is at least 10,000
>     times smaller than the load on the data plane (so that each
>     time a packet goes to the control plane, on the average there
>     are 10,000 or more packets that will be forwarded via the data
>     plane before the next packet needs to go to the control plane)
>
> I am guessing that the answer is either (5), or perhaps (3), or
>   (6) You are going to deploy LISP closer to the edge (and NOT
>     in the PE router).

If I can paraphrase you, Ross, doing packet forwarding at 10Mpps  
doesn't scare me.

;-)

Tony

_______________________________________________
RAM mailing list
[email-masked]
https://www1.ietf.org/mailman/listinfo/ram

---

From: Markus Stenberg <[email-masked]>
Date: Mon, 02 Apr 2007 11:46:50 +0900

Dino Farinacci <[email-masked]> writes:
>> d) some poor firewall or ACL filters out the ICMP message at the ICMP
>>    originator's domain, in transit, or at the recipient's domain?
> The protocol needs feedback. If you choose UDP to return the mapping
> or error message, those could get filtered as well. There is no magic
> here, you have to fix the filters.
>
> If we let these things get in the way, nothing will be able to get
> deployed.

So far most of the solutions IETF has been working on has been routing
around the damage (NATs -> NAT-traversal, ICMPs lost -> don't rely on them)
rather than mandating policy. Mandating a policy (BCP-38) hasn't had much
of a success, unfortunately.

>> e) the ICMP error gets lost in the network (transient routing loop,
>>    congestion, etc.).  ICMP delivery is after all unreliable.
>
> Oh well, would you rather have a TCP connection between active ITR/
> ETR pairs, I don't think so.
>
> ARP requests and replies are unreliable.
> DNS requests and replies are unreliable.
> SNMP packets are unreliable.
>
> We have lived with this. The Internet is best-effort and I think it
> does a good job of being "best".

I think you're nitpicking; ICMP is about as reliable communication media as
sending the messages using RFC1149 in modern internet - certainly,
ping/traceroute work for some definition of works, but PMTU-related seems
to break quite frequently. And do you really believe you can get everyone
to fix their filters to get 'incrementially' deployed solution working as
well as current state?

This sounds like a non-starter to me. The 'unreliable' by virtue of
unreliable packets is orders of magnitude more reliable than ICMP in the
modern internet. (rate limiting, filtering, etc, all aimed at dropping
excess ICMP traffic.)

>> While participating in Transport Area working groups (e.g., TCPM)
>> I've had an.. opportunity to hear rants about ICMP, for example,
>> that it should be considered completely optional and anything that
>> requires ICMP to work is broken; at best, ICMP should be an
>> optimization mechanism.
> What would you suggest as an alternative that won't have the same
> issues?

If you're designing a new protocol, might as well get a new protocol number
for it (if you're really not afraid of filters), or use UDP (if you are).

>> I'm not sure if I completely agree with these, but for sure I
>> wouldn't built a system that had a very strong requirement for ICMP
>> error generation and propagation.
> The messages were defined originally to be of value. Yes, they have
> been abused over the years, but the intent of the original designers
> was to help the network. The LISP authors plan to continue to use ICMP.

Well, then all I can tell you is, good luck, you will need it. :-)

Cheers,

-Markus

---

From: Dino Farinacci <[email-masked]>
Date: Sun, 1 Apr 2007 23:12:48 -0700

> I'm not yet convinced that this is really a requirement.  Consider  
> what you're asking: the ability to TE on a per-site basis, given  
> that traffic is already assigned to PA prefixes.  Typically in the  
> past, ISPs have wanted to do TE within another providers prefix,  
> and thus we have more specifics.  This is incredibly important for  
> managing peering ratios and the like.  However, applying TE on  
> multi-homed PI sites seems like it would be much less important  
> requirement.

For the LISP model, packets that get to a TE-ITR do not have PI  
addresses in the outer DA. Those have been buried by having a ITR  
from the source site encapsulate putting locators in the outer DA.

Dino

---

From: Dino Farinacci <[email-masked]>
Date: Sun, 1 Apr 2007 23:19:48 -0700

> modern internet. (rate limiting, filtering, etc, all aimed at dropping

There is reliability in the system. The next non-rate-limited data  
packet will be responded to with an ICMP transmission by the ETR.

> for it (if you're really not afraid of filters), or use UDP (if you  

This is a simple fix. We can do this.

> Well, then all I can tell you is, good luck, you will need it. :-)

Let me repeat, this is a prototype, the end result could look very  
different.

Dino

---

From: Brian E Carpenter <[email-masked]>
Date: Mon, 02 Apr 2007 10:32:20 +0200

On 2007-03-31 10:13, Dino Farinacci wrote:
...
> The Internet is best-effort, we are not trying to make datagram delivery 
> more reliable. If a EID mapping has multiple locators and the ETR 
> indicates the priority of each locator is the same, the ITR can 
> load-split traffic among the locators.

That raises the spectre of out-of-order delivery, which can be very
painful for the transport layer.

> The protocol needs feedback. If you choose UDP to return the mapping or 
> error message, those could get filtered as well. There is no magic here, 
> you have to fix the filters.

Well, I have to ask the same question I recently asked about shim6
error messages: will lost ICMP packets break the protocol? If yes,
then I think that signalling in-band (i.e. in packets that are part
of the target session) is much preferable. "Fix the filters" isn't
a deployable feature of LISP, IMHO, because different people control the
filters.

    Brian

---

From: Bruce Curtis <[email-masked]>
Date: Mon, 2 Apr 2007 10:35:00 -0500

On Apr 2, 2007, at 3:32 AM, Brian E Carpenter wrote:

>> The Internet is best-effort, we are not trying to make datagram  
>> delivery more reliable. If a EID mapping has multiple locators and  
>> the ETR indicates the priority of each locator is the same, the  
>> ITR can load-split traffic among the locators.
>

   I think it would be an interesting approach to load balancing or  
TE if the packets contained the multiple addresses of the destination  
plus the ratio the destination would like sent to each address.  For  
example 25% to address A and 75% to address B.  Of course for real  
time data like voice you could have a Don't Load Balance Bit.  ( I  
think there was mention of something similar for STCP in the  
Architecture-Discuss list).

---
Bruce Curtis                         [email-masked]
Certified NetAnalyst II                701-231-8527
North Dakota State University

---

From: Dino Farinacci <[email-masked]>
Date: Mon, 2 Apr 2007 10:33:38 -0700

> painful for the transport layer.

A traditional tradeoff between route-based versus flow-based  
forwarding. And the decent middle ground adopted today is hashing.

So for an EID-prefix match, you hash on the 4-tuple say, and then  
pick a Locator (and stick with it) from the Locator-set.

Brian, don't worry, be happy.  ;-)

>> The protocol needs feedback. If you choose UDP to return the  
>> mapping or error message, those could get filtered as well. There  
>> is no magic here, you have to fix the filters.
>

Then you forward into black-holes. That tends to get people's  
attention to get the problem fixed.

There is no silver bullet here Brian. You know that, you've been  
doing this too long as well.  ;-)

Dino

---

From: Peter Schoenmaker <[email-masked]>
Date: Mon, 2 Apr 2007 14:25:06 -0400

On Mar 31, 2007, at 12:23 PM, Tony Li wrote:

>
> On Mar 31, 2007, at 2:22 AM, marcelo bagnulo braun wrote:
>
>> i think this is one fundamental part of the puzzle that is lacking  
>> in all available solutions
>
>
> Am I confused?  Any operators in the forum, please speak up...

For the case of multihomed end sites the weighting in the mapping  
lookup should be sufficient for TE.  The requirements are fairly  
simple.  For the case of TE between ISPs, the TE requirements are  
more complicated.  Hot potato vs cold potato.  Which adjacent AS to  
prefer, and which link to an adjacent as to use.

peter

---

From: Brian E Carpenter <[email-masked]>
Date: Tue, 03 Apr 2007 13:22:58 +0200

On 2007-04-02 19:33, Dino Farinacci wrote:
>>> the ETR indicates the priority of each locator is the same, the ITR 
>>> can load-split traffic among the locators.
>>
> A traditional tradeoff between route-based versus flow-based forwarding. 
> And the decent middle ground adopted today is hashing.
> 
> So for an EID-prefix match, you hash on the 4-tuple say, and then pick a 
> Locator (and stick with it) from the Locator-set.

Yes, any trick that allows reasonable in-orderness per flow is fine.
> Brian, don't worry, be happy.  ;-)

I try :-)

> Then you forward into black-holes. That tends to get people's attention 
> to get the problem fixed.

But that's my point. This sort of thing *doesn't* get the attention of
the right people, because it affects Joe Citizen, whose only recourse
is an ISP (or corporate) help desk. You know how devoted they are to
fixing 3rd party connectivity issues. I believe we have an iceberg
shaped problem today of  black holes caused by NAT nastiness, which
are never identified as such, and I don't think we should create
more of that sort of thing.

> 
> There is no silver bullet here Brian. You know that, you've been doing 
> this too long as well.  ;-)

Yes.

     Brian

---

From: Jari Arkko <[email-masked]>
Date: Tue, 03 Apr 2007 22:16:44 +0300

Peter,

> simple.  For the case of TE between ISPs, the TE requirements are more
> complicated.  Hot potato vs cold potato.  Which adjacent AS to prefer,
> and which link to an adjacent as to use.
Let me ask for a clarification. When you say that the TE requirements
are simple for end sites, are you saying that from the point of view
of a provider who needs to apply TE to that site? Or from the point
of view of the end-site network manager? What about the mapping
weights -- is a static mapping sufficient, or does it have to be
dynamic?

Jari

---

From: Peter Schoenmaker <[email-masked]>
Date: Fri, 6 Apr 2007 13:03:38 -0400

On Apr 3, 2007, at 3:16 PM, Jari Arkko wrote:

>> more

I'm talking from what i perceive as the TE requirements of the end- 
site network manager.  From my view there are two types of multi- 
homed customers.  The simple one where they have one location with  
connections to multiple providers, the other is where a multi-site  
customer network connects to multiple providers in multiple  
locations.  The complex multihoming customer actually appears more as  
an ISP rather than a multihomed customer.

I think for the simple multihomed customer the default configuration  
would be static mapping which should cover most needs.  A large  
benefit of a mapping solution is that it would allow people to  
develop a dynamic system to meet their needs.

peter

---

From: marcelo bagnulo braun <[email-masked]>
Date: Sat, 7 Apr 2007 17:58:08 +0200

sorry for the delayed reply...

El 01/04/2007, a las 1:18, Tony Li escribió:

>> but this seems to be the only difference from the operator 
>> perspective, between a PI-BGP based multihoming solution and a 
>> multihoming solution based on PA and an host based mechanism like HIP 
>> or shim.
>
>
> Not necessarily.  Consider an ITR/ETR co-located with a PE router.  
> That's very interesting from an operator perspective as they can then 
> effectively change locators for a site without site intervention.
>

i am not sure how this would work...

are you considering the case of a multihomed site connected to two or 
more different ISPs?
because if this is the case, it seems a bit strange to me that a end 
site will allow one of its isps to select the other ISP to send traffic 
(by selecting the other ISP prefix/locator)

i mean, i think that allowing a ISP to select the other ISP locator may 
not be such a good idea and seems a bit off limits of what a ISP should 
be able to control on its client's equipment...

am i missing something?

>
>> since the ISPs claim that there is some TE functionalities missing, 
>> it zould be important to understand what are those so ze can actually 
>> provide those...
>
>
> I agree.   Moreover, it's important that we confirm that the features 
> that we are providing are those that are beneficial.
>

agree

but it seems that so far, we haven't managed to get the operational 
community to describe what exact TE capabilities they need (except from 
J. Schiller initial effort a year ago, but that didn't end up 
documented in a draft AFAICT...)

regards, marcelo

---

From: Tony Li <[email-masked]>
Date: Sat, 7 Apr 2007 13:52:41 -0700

>> Consider an ITR/ETR co-located with a PE router.  That's very  
>> interesting from an operator perspective as they can then  
> are you considering the case of a multihomed site connected to two  
> or more different ISPs?

Nope, it's interesting even for a singly homed site.

> because if this is the case, it seems a bit strange to me that a  
> end site will allow one of its isps to select the other ISP to send  
> traffic (by selecting the other ISP prefix/locator)
>
> i mean, i think that allowing a ISP to select the other ISP locator  
> may not be such a good idea and seems a bit off limits of what a  
> ISP should be able to control on its client's equipment...

Many sites want to outsource as much as possible.  Letting/asking/ 
paying for their ISP to manage something for the site is common.   
Consider PPVPNs, for example.

Of course, it is optional.  Having the ITR be the CE box (or one of  
many) is also an viable alternative.

Tony

---

From: marcelo bagnulo braun <[email-masked]>
Date: Sun, 8 Apr 2007 10:41:06 +0200

> Nope, it's interesting even for a singly homed site.

so, a single homed site with multiple locators? what would be the 
motivation for that? do you think this would be a common case?

>> site will allow one of its isps to select the other ISP to send 
>> may not be such a good idea and seems a bit off limits of what a ISP 
>> should be able to control on its client's equipment...
>
>
> Many sites want to outsource as much as possible.  
> Letting/asking/paying for their ISP to manage something for the site 
> is common.

right, but as long there is no conflict of interests. letting one ISP 
to forward the traffic through the alternative ISP could generate such 
conflict of interests... I guess it depends on the pricing structure. I 
mean, if the end siFrom [email-masked] Sun Apr 08 04:56:06 2007
From: marcelo bagnulo braun <[email-masked]>
Subject: Re: [RAM] Comment on draft-farinacci-lisp-00.txt (LISP)
Date: Sun, 8 Apr 2007 10:41:06 +0200
To: Tony Li <[email-masked]>
Cc: [email-masked]

> Nope, it's interesting even for a singly homed site.

so, a single homed site with multiple locators? what would be the 
motivation for that? do you think this would be a common case?

>> site will allow one of its isps to select the other ISP to send 
>> may not be such a good idea and seems a bit off limits of what a ISP 
>> should be able to control on its client's equipment...
>
>
> Many sites want to outsource as much as possible.  
> Letting/asking/paying for their ISP to manage something for the site 
> is common.

right, but as long there is no conflict of interests. letting one ISP 
to forward the traffic through the alternative ISP could generate such 
conflict of interests... I guess it depends on the pricing structure. I 
mean, if the end site pays its isp by load, then i guess that maybe the 
conflict of interests could be avoided, since the incentives seem to 
lead isps to attract traffic, otoh, if the end site pays flat rate, i 
can see an incentive for the isp to divert as much traffic as possible 
to the alternative isp.

regards, marcelo

---

From: Tony Li <[email-masked]>
Date: Sun, 8 Apr 2007 09:03:48 -0700

>>> are you considering the case of a multihomed site connected to  
>>> two or more different ISPs?
>>

No, a single locator is still interesting.  There are numerous cases  
where, as an ISP grows, they would like to renumber a customer and  
can't today.  Typical examples are ISPs that did initial address  
allocations without regard to future deployment of IGP areas, and  
ISPs that acquired other, geographically overlapping ISPs.

A Loc/ID split helps the site have provider independence, and it also  
helps the ISP with internal aggregation.

> right, but as long there is no conflict of interests. letting one  
> ISP to forward the traffic through the alternative ISP could  
> generate such conflict of interests... I guess it depends on the  
> pricing structure. I mean, if the end site pays its isp by load,  
> then i guess that maybe the conflict of interests could be avoided,  
> since the incentives seem to lead isps to attract traffic, otoh, if  
> the end site pays flat rate, i can see an incentive for the isp to  
> divert as much traffic as possible to the alternative isp.

You mean an ISP might use traffic engineering to optimize its  
situation?  I'm shocked, shocked that you would suggest such a  
thing!  ;-) ;-) ;-)

Tony

---

From: marcelo bagnulo braun <[email-masked]>
Date: Sun, 8 Apr 2007 20:18:18 +0200

El 08/04/2007, a las 18:03, Tony Li escribió:

> allocations without regard to future deployment of IGP areas, and ISPs 
> that acquired other, geographically overlapping ISPs.
>

I see what you mean and it makes sense to me

but a couple of comments: i) this is not really related to TE, right? 
(or at least to inter domain TE) ii) you are assuming provider 
independent identifiers?

>> to forward the traffic through the alternative ISP could generate 
>> such conflict of interests... I guess it depends on the pricing 
>> structure. I mean, if the end site pays its isp by load, then i guess 
>> that maybe the conflict of interests could be avoided, since the 
>> incentives seem to lead isps to attract traffic, otoh, if the end 
>> site pays flat rate, i can see an incentive for the isp to divert as 
>> much traffic as possible to the alternative isp.
>
> situation?  I'm shocked, shocked that you would suggest such a thing!  
> ;-) ;-) ;-)

:-)
of course the ISP may want to do that

what i would be surprised is that the customer would allow it... i 
mean, the client does have to let ISP1 know about ISP2 and its prefix, 
in order to let ISP1 to divert the traffic through ISP2, (this would 
imply for someone to configure both prefixes in the same box, right?) 
and this doesn't makes sense to me so far.

Also you could end up in a strange/unstable situation where ISP1 push 
all the traffic to ISP2 and ISP2 pushes the traffic to ISP1, in order 
to.

So summarizing, i do see a benefit w.r.t. to PI to allow a single box 
to contain all the PA locator prefix information and even allowing the 
ISP to manage that box, so that intra ISP renumbering events are 
transparent to the customer.

Still i don't see clear that allowing the ISP to manage that box in a 
multihoming scenario with multiple ISPs in order to perform TE makes 
sense from the client p.o.v.

Regards, marcelo

---

From: Tony Li <[email-masked]>
Date: Sun, 8 Apr 2007 15:26:11 -0700

> but a couple of comments: i) this is not really related to TE,  
> right? (or at least to inter domain TE) ii) you are assuming  
> provider independent identifiers?

Correct, this has nothing to do with TE.  Yes, this is assuming that  
identifiers are provider independent.  Identifiers that were provider  
dependent are not useful in the least.

> mean, the client does have to let ISP1 know about ISP2 and its  
> prefix, in order to let ISP1 to divert the traffic through ISP2,  
> (this would imply for someone to configure both prefixes in the  
> same box, right?) and this doesn't makes sense to me so far.

That would assume that ISP1 was intentionally diverting the traffic.   
Any such action would be visible on the other side of the net.  All  
it would take would be a looking-glass type mechanism to make this  
plainly visible.

Note that this situation already exists today.  All that ISP1 has to  
do is to AS prepend the customer's PI prefix to divert traffic.   
Thus, either (a) the looking glass check stops the ISP from being  
dishonest or (b) he's not such an Evil Greedy Bastard after all.  ;-)

> Also you could end up in a strange/unstable situation where ISP1  
> push all the traffic to ISP2 and ISP2 pushes the traffic to ISP1,  
> in order to.

Kinda like AS prepend wars?

> So summarizing, i do see a benefit w.r.t. to PI to allow a single  
> box to contain all the PA locator prefix information and even  
> allowing the ISP to manage that box, so that intra ISP renumbering  
> events are transparent to the customer.
>
> Still i don't see clear that allowing the ISP to manage that box in  
> a multihoming scenario with multiple ISPs in order to perform TE  
> makes sense from the client p.o.v.

Again, the enterprise is strongly motivated to outsource, if  
possible.  As discussed, there are ample checks and balances that  
would curb any ungentlemanly behavior.

Tony

---

From: Ted Seely <[email-masked]>
Date: Mon, 9 Apr 2007 11:55:34 -0400 (EDT)

> sense from the client p.o.v.

hmmmmm.... last i checked that was Internaps buisiness model, and they
seemed to have a healthy number of customers who wanted exactly that.

I don't see why in the foreseeable future that would change, and that is
the beauty of the outsource model.

-ted

---

From: Yakov Rekhter <[email-masked]>
Date: Sun, 15 Apr 2007 05:59:19 -0700

Tony,

> If I can paraphrase you, Ross, doing packet forwarding at 10Mpps  

How many EID-to-RLOC mappings ICMP messages per second an ITR/ETR be
required to originate/receive ?

Yakov.

---

From: Eliot Lear <[email-masked]>
Date: Sun, 15 Apr 2007 18:32:31 +0200

> required to originate/receive ?

Doesn't that very much depend where the ITR/ETR sits?  And isn't the 
point of an experiment to find out?

Eliot

---

From: Tony Li <[email-masked]>
Date: Sun, 15 Apr 2007 14:18:11 -0700

> required to originate/receive ?

Yakov,

I think that you're confusing the encap rate with the mapping rate.

Assuming that we eventually get the mapping mechanism correct (which  
admittedly may require LISP version N for arbitrarily large N ;-),  
then some low message rate, asymptotically approaching 0 pps should  
be required.  The precedent here is the overhead required by a site's  
local caching DNS server.

Tony

---

From: Yakov Rekhter <[email-masked]>
Date: Mon, 16 Apr 2007 06:32:37 -0700

Tony,

> I think that you're confusing the encap rate with the mapping rate.

Incorrect.
  
> be required.  

My question was in the context of LISP 1.5.

> The precedent here is the overhead required by a site's  

The analogy does not hold. This is because DNS does not provide any
information about whether a particular host is reachable or not.
In contrast, in LISP the ICMP messages do provide
reachability/unreachability information about RLOCs.

Yakov.

---

From: Olivier Bonaventure <[email-masked]>
Date: Tue, 17 Apr 2007 16:30:23 +0200

Yakov,

> required to originate/receive ?

We did a related study several years ago based on Netflow traces
collected in 1999. We used Netflow traces from two different ISPs, one
was a dialup ISP (about 6 Mbps of peak traffic) and the other a research
ISP (about 80 Mbps of peak traffic). At that time, the BGP tables in the
DFZ contained 70000 prefixes and we used the trace to look at the number
of tunnels that would have to be established towards the prefixes that
are actually receiving packets from the studied ISPs.

For the dialup ISP, there were about 2000 tunnels active (assuming that
a tunnel was active during every minute in which a packet was sent
towards the prefix) on average. For the research ISP, there were about
12000 active tunnels. The average churn for the tunnels was of about
10%. If we had tested a solution like LISP at that time, the dialup ISP
would have received on average 200 ICMP messages per minute (with peaks
of 900). For the research ISP, the average was 1100 ICMP messages per
minute and the peak about 1500 ICMP per minute.

Additional information about the traffic traces and the results may be
found in the (old) paper below

[UB00] 	S. Uhlig and O. Bonaventure. On the cost of using MPLS for
interdomain traffic. In Quality of Future Internet Services (QoFIS
2000), September 2000. Available from
http://totem.info.ucl.ac.be/publications.html

We expect that the Internet traffic has changed a lot since 1999 with
the introduction of peer-to-peer and worms. These two factors probably
increase the number tunnels that need to be established and maintained.

Olivier

-- 
CSE Dept. UCL, Belgium - http://www.info.ucl.ac.be/~obo