Subject: [lisp] comments on draft-ietf-lisp-06.txt (part 1)

From: Yakov Rekhter <[email-masked]>
Date: Mon, 19 Apr 2010 13:48:34 -0700

Folks,

John Scudder, John Drake, and myself produced a list of comments
on draft-ietf-lisp-06.txt. The comments are split into two parts:
(1) comments that are not specific to a particular section, and
(2) comments that are specific to a particular section. This
e-mail contains (1).

Yakov.
-----------------------------------------------------------------
Comment 1:

Does the target audience for LISP include those who already are
using PI addresses for multihoming ? If yes, then what does it
offer to this audience above and beyond of what they have now ?

Does the target audience for LISP include those who already are
using NAT ? If yes, then what does it offer to this audience above 
and beyond of what they have now ?

--------------------------------------------------------------------
Comment 2:

There is no discussion of cache thrashing by hosts within a site which
(either for some legitimate reason, or because of infection by a virus
or worm) send packets to a large span of addresses.  In addition to
cache thrashing, this would create huge amounts of LISP control plane
traffic. Even if rate limiting on the ITR helped prevent this, the rate
limiting would have to be quite sophisticated (per host) to prevent a
badly-behaved host from hurting other hosts at its site.  Also, it
assumes that sending packets to a large span of addresses is bad
behavior. This should be documented as a limitation compared to the 
current routing model.

--------------------------------------------------------------------
Comment 3:

The draft leaves a number of important aspects of the Map-Reply
open, or only implied rather than clearly stated.  These should be
stated in one place, clearly. Ambiguous areas include:

- Is there a requirement that all entities that return Map-Replies 
  for a given EID return the same contents?
  - If yes, then what are the consequences if two entities do return 
    Map-Replies with different contents?
  - If no, then
     - What content is allowed to vary?  (Locator Status Bits?)
     - What content must be invariant?  (locator-set?)

- If for a given EID its locator-set returned is required to be invariant, 
  how is this achieved ?  
  - Must the locator-set be manually configured on every entity that 
    returns Map-Replies for that EID?
  - Some other way?

- Under what circumstances might an EID-to-RLOC mapping change?  
  - Can it change only as a result of provisioning/configuration ?
  - Or might it change as a reaction to the network's operational state?  
    (E.g. temporary loss of reachability from a given ETR to a given EID.)

--------------------------------------------------------------------
Comment 4:

Suppose a certain EID's mapping entry returned at some point in the
past in the Map-Reply by ETR2 lists the RLOCs of two ETRs, ETR1 and
ETR2. Suppose that later on due to connectivity issues within the
site, ETR1 is able to reach the EID, but ETR2 is not. Is there some
means by which traffic is expected to reliably reach the EID in
this case?  What is it? (Keep in mind that the same RLOC of ETR2
may also be used to reach some other EID2, and ETR2 may have fine
connectivity to EID2.)

--------------------------------------------------------------------
Comment 5:

Consider the following example:

                  ETR1
   ITR      
                  ETR2

Originally both ETR1 and ETR2 claim to be RLOCs for EIDs 10.1.1/24
and 10.1/16. So, when ITR sends Map-Request to ETR1 for 10.1.5.1,
ETR1 returns both 10.1/16 and 10.1.1/24 as EID prefixes, with ETR1
and ETR2 as RLOCs for these EIDs.

Assume that at some later point ETR1 is no longer an RLOC for
10.1.1/24, but still is an RLOC for 10.1/16, while ETR2 still is
an RLOC for both 10.1.1/24 and 10.1/16. Would ETR1 be required to
inform ITR that it is no longer an RLOC for 10.1.1/24 ?

If not then assume that later on a site behind the ITR originates
a packet for 10.1.1.1. Assume that the ITR selects ETR1 as the best
RLOC for 10.1.1/24. So, the ITR sends the packet to ETR1.  How would
ITR ultimately decide to stop sending the packets destined to
10.1.1.1 to ETR1 and instead switch to ETR2 ?

If ETR1 be required to inform ITR that it is no longer an RLOC for
10.1.1/24, then what mechanisms, other than the Solicited-Map-Request,
ETR1 could use ? If the only mechanism available is the
Solicited-Map-Request, then this introduces its own set of problems,
as follows.

Since according to 6.5.2 the ETR "will solicit Map-Request from
sites it is currently sending encapsulated data to, and only from
those sites", how would that help to update the mapping if the ITR
that has the old (incorrect) mapping is not the xTR to which the ETR
sends the encapsulated data ?

Withdraw of a single EID prefix on the ETR may require this ETR to
send a whole bunch of SMRs messages, in response receive a bunch
of Map-Requests, and then send Map-Replies. And while doing this
all the data traffic for that EID prefix is dropped on the floor.
This has negative implications on the connectivity restoration time.

Moreover, while the stated goal of Solicit-Map-Request is to control
the rate an ETR receives Map-Requests messages, in the above scenario
in order to reduce the connectivity restoration time the ETR has
to send all these Solicit-Map-Requests messages as soon as ETR1
determines that it is no longer an RLOC for 10.1.1/24, which in
turn would mean that it would receive a burst of Map-Requests causing
potential congestion in the control plane, which could result in
some of these Map-Requests being dropped. Would the ETR be required
to re-send Solicit-Map-Request if it does not receive a corresponding
Map-Request from the ETR to which it sent Solicit-Map-Request before ?
And even if Map-Request is not dropped, the corresponding Map-Reply
could get dropped in transit. What would happen in this case ?
------------------------------------------------------------------------
Comment 6:

ITR failure and its implications on connectivity restoration time

Consider a scenario where a failure (e.g., ITR failure) would result
in shifting a large fraction or all the flows that used to go through
one ITR to another ITR. Note that in this scenario the arrival rate
of the flows at the new ITR is determined by the total number of
flows being shifted (e.g., in the case of an ITR failure it is the
number of flows that used to go through the old ITR), and not by
the steady state arrival rate of flows. Therefore, in this scenario
the rate with which the new ITR has to resolve EID-RLOC mapping is
determined not by the arrival rate of flows in the steady state,
but by the total number of flows being moved to the new ITR, and
the latter may be significantly higher than the former. 

Even if we assume that all flows are TCP-based (no UDP-based flows
at all), then while the arrival rate of the first few packets at
the beginning of such flows is determined by the SYN retransmission
time (and thus could be fairly low), this is not the case for the
arrival rate of the packets in the middle of such flows, which could
be Mb/sec, or even Gb/sec. Thus the amount of dropped packets (if
ITR drops packets while resolving EID-RLOC mapping) or buffered
packets (if ITR buffers packets while resolving EID-RLOC mapping)
in a situation where a failure results in a whole bunch of existing
flows being rerouted to a new ITR is going to me much higher than
in the steady state. Note than none of this occurs with today's
routing. That means that in the presence of ITR failure LISP would
negatively impact connectivity restoration time relative to what
we have today, as well as the service availability.

An approach, where instead of dropping packets, an implementation
would queue packets would require additional memory on the router,
which at the minimum would increase the cost (and therefore the
price) of the router. Moreover, in the scenario where a failure of
one ITR would result in shifting all the flows that used to go
through that ITR to another ITR, if the number of flows is large
and/or flows are of high bandwidth it may be unfeasible to queue
all these packets on the new ITR.

One can not claim that the problem of how to handle data while
resolving EID-RLOC mapping is a local problem for which there are
local incremental fixes, as there are no practical solutions to
this problem on the table at all, and thus one can not claim that
such problem could be solved by local incremental fixes. Since this
problem is likely to manifest itself most visibly at large scale
deployment, it would be inappropriate to wait with solving this
problem until large scale deployment - this problem needs to be
solved before any large scale deployment of LISP, or any large scale
deployment of LISP should be put on hold until a practical solution
to this problem is developed.

------------------------------------------------------------------------
Comment 7:

ETR failure and its implications on connectivity restoration time

Note that in today's routing global BGP convergence is NOT required
for connectivity restoration. In other words, IP connectivity
restoration takes less time that what it takes to propagate fault
information throughout the Internet (or for that matter even
throughout a site). With LISP connectivity restoration requires to
propagate fault all the way to the ITRs. In addition, if one relies
on Loc-Status-Bits to determine that a particular RLOC is down,
connectivity restoration would also require to propagate fault
information from one ITR to another. In some case this may not be
feasible (e.g., ETRs are PEs). And even in the cases where it is
feasible, that is likely to increase connectivity restoration time.

---

From: Dino Farinacci <[email-masked]>
Date: Tue, 20 Apr 2010 00:39:11 -0700

> (2) comments that are specific to a particular section. This

Yakov/John/John, thanks for your comments. I am glad that Juniper is  
getting involved with LISP. Other equipment vendors like Ericsson, Hua- 
Wei, Arista, and XKL, to name a few, have been involved as well.

We are in the process of making changes to -07 and have to make a call  
on how many changes we put in. But what we cannot put into -07, we  
will definitely put into -08, which should be published before the  
next IETF. Ditto for the lengthy comments Dimitri sent last week.

We are just trying to treat high priority changes with expediency and  
others that may require more data or additional testing come next. So  
don't take us not reflecting a comment as discounting it. But please  
verify in -08 that we have addressed your comments.

See responses to your comments inline.

Yes, it does include these sites. What is offered is ingress control  
at the site. And the user does not have to run a more complicated BGP  
protocol on their CPE routers.

Outsourcing costs to manage routers is soaring and isn't cost  
justified by branch offices who want to become multi-homed.

> using NAT ? If yes, then what does it offer to this audience above

Yes, it does. It is very difficult today to do multi-homing with  
ingress traffic engineering when you have NATs.

We have been validated by customers that it is desirable to use global  
addresses for better policy and identification of users.

> and beyond of what they have now ?

We have text indicating how do to cache management. We are testing  
implementations on how to aggressively delete entries not being used  
and how to damp and rate-limit map-requests.

We do have a lot of experience with this because the 65 node network  
we have deployed has a half-of-dozen PITRs deployed. The Internet has  
a common pulse of address scans 24x7, so we see how the cache is being  
maintained on those PITRs. It has been useful to tune the  
implementation so we can put text in the drafts to describe how to  
deal with this.

It really hasn't been a problem.

> limiting would have to be quite sophisticated (per host) to prevent a

It's not sophisticated, it is granular, if it needs to be.

> badly-behaved host from hurting other hosts at its site.  Also, it

On the contrary, the ITR is the one that can block a malicious host at  
a LISP site to reduce the traffic load further on the path. And the  
state is local to the sources on at the site, rather than a much  
larger set.

With ETR to ITR feedback, we can actually source-quench rogue sources.

> behavior. This should be documented as a limitation compared to the

Yes, it is stated, the ETRs should return the same contents. When I  
say "contents" here, I mean the locator-set has the same list of  
locator addresses. See below for changing priority/weight values per  
locator address.

>    Map-Replies with different contents?

Not very severe, the ITRs that cache one versus the other just get  
different answers and use what they have cached. If the LSBs are  
inconsistent, then one just avoids using some of the locators. It just  
means that traffic is not load balanced even if the destination site  
might want it to be. But since the configuration error is the fault of  
their own, they are the ones that are affected.

>     - What content must be invariant?  (locator-set?)

We envision for policy reasons that the Map-Reply contents (not the  
locator-set itself but the priorities and weights associated with each  
locator) may vary based on the source of the Map-Request or the  
destination address the Map-Request is for, that priorities/weights  
may differ. But we encourage that the same policy be configured  
consistently among all ETRs that would be answering a Map-Request.

> - If for a given EID its locator-set returned is required to be  

It is not required but desirable for network management reasons.

>    returns Map-Replies for that EID?

Yes, we recommend each ETR either is connected directly to the ALT or  
registers to a Map-Server, but if they do not they won't get Map- 
Requests so they don't or can't send Map-Replies. We encourage all  
ETRs to accept Map-Requests so the control-plane load can be balanced  
across all the CPUs of the various ETRs.

>  - Can it change only as a result of provisioning/configuration ?

Yes. Usually this case.

>    (E.g. temporary loss of reachability from a given ETR to a given  

No, as discussed in the spec, we keep locator reachability out of the  
mapping database system. Thereby if a locator goes down or out of  
service, *it is not removed* from the locator-set.

Yes, ETR1 can tell remote sites by setting its own LSB to 0. ETR2 can  
do the same for ETR1's LSB.

Right.

> inform ITR that it is no longer an RLOC for 10.1.1/24 ?

Either ETR1 or ETR2 can use versioning numbering (new in -07) or SMRs  
(Solicit Map-Requests) to tell cachers that the locator-set is  
changing for either EID-prefix. In this case, both ETR1 and ETR2 have  
been reconfigured to have a locator-set for 10.1.1.0/24 to be only ETR2.

> sends the encapsulated data ?

If RLOC-probing is in use from the ITRs that have enabled it, they  
will get Map-Replies with the new locator-set. If no RLOC-probe reply  
comes back, then the ITR stops encapsulating to that locator.

But when using SMRs, the other xTRs will inform the cachers.

> of Map-Requests, and then send Map-Replies. And while doing this

A bunch is rate-limited to the top talkers, the ones that need  
updating sooner rather than later.

> This has negative implications on the connectivity restoration time.

There is also a method to set the LSB bit for a removed locator. That  
can be done in the data-plane so the restoration is equal to the  
amount of time the data-plane processor in the xTR can tell the  
control-plane processor. We are talking 10 to 100s of microseocnds  
versus network latency time (10s of milliseconds) or protocol timeout  
ranges.

> potential congestion in the control plane, which could result in

Map-Requests will be sent by multiple ITRs on the ALT that has  
multiple paths to the Map-Servers that are advertising the EID-prefix.  
So the load is distributed across the ALT. Also, the Map-Servers  
distribute the load across the ETRs registering. That makes use of all  
the resources of the mapping database system.

Plus any implementation that blasts out packets can expect to overload  
its output queue first and foremost, so it needs to clock its output.  
And that will help the network as well.

> could get dropped in transit. What would happen in this case ?

Have a look at the map-versioning specification. This can help the  
situation especially in the case of unidirectional traffic (i.e. PITR  
to ETR flows).

Just like when a new ITR comes up. It's the same case.

> we have today, as well as the service availability.

If you are really worried about this case, there are couple solutions  
we have not documented:

(1) we can pre-populate map-cache entries for popular sites. This is  
similar to how browsers ship with factory-installed Certificates.

(2) The Map-Request/Map-Reply packet format is flexible enough for an  
ITR to ask another ITR at its own site for what it has cached. It  
*could* store those entries before the flows come to it. This is a  
memory/state tradeoff.

> would queue packets would require additional memory on the router,

Many customers have said dropping is better than queuing, because  
predictable latency is more important.

We do not suggest additional queuing in routers to support LISP.

> all these packets on the new ITR.

We have implemented aLISP in an existing production router with simply  
a software upgrade change. It sits at the CPE since we believe that is  
the sweet-spot for site control. No new memory or bandwidth  
requirements.

> such problem could be solved by local incremental fixes. Since this

Well, we have to leave some creativity for the manufacturer.  ;-)

> problem until large scale deployment - this problem needs to be

You don't need large scale deployment. You can prove it with 2 LISP  
sites, each storing /32 EID-prefixes for the other. We don't encourage  
this but we can get good data points by this example.

The fact that the scan attacks are constant on the Internet and the  
boxes are being held together is saying the problem isn't as bad you  
may think by just reading documents.

> to this problem is developed.

And that is what we are doing.

Since there isn't very good aggregation of prefixes in BGP, route- 
updates tend to go farther than one would desire (especially when  
there are PI prefixes all over the place and just increasing). There  
is also queuing delays, processing delays, and TCP delays in each node  
that has to process the route change.

In the LISP case, the ITR hardware is programmed with the new LSB  
settings and every router on the path is hardware switching this data  
packet to the ETR. So the cost in processing is *only* at the ETR. So  
it is much faster than in the BGP case. And in the LISP case, the  
source site gets the granularity it needs versus a BGP aggregate that  
doesn't tell the source site anything about the *real* reachability of  
the destination.

Please note, just because you have a BGP route in your routing table  
does not mean you can reach the destination.

We will get you responses to your part 2 comments by end of week.

Thanks again,
Dino/Dave/Vince/Darrel

> _______________________________________________
> lisp mailing list
> [email-masked]
> https://www.ietf.org/mailman/listinfo/lisp

---

From: John Scudder <[email-masked]>
Date: Tue, 20 Apr 2010 10:04:27 -0700

Hi Dino,

Just replying to one point here.  Other replies as time allows.

On Apr 20, 2010, at 3:39 AM, Dino Farinacci wrote:
[...]
> Yes, it is stated, the ETRs should return the same contents. When I

I've only read the spec a few times so I may have missed where this is spelled out.  Would you mind citing where this is stated in the spec?

> Yes,

Is that "yes" to the "must be manually configured" question?  I would have assumed so but you go on to talk about connecting to the ALT and such, which would appear to be an unrelated (or tangentially related) topic.

> No,

OK...

> as discussed in the spec, we keep locator reachability out of the

... but note that the parenthetical question had to do with *EID* reachability, not locator reachability.

--John

---

From: Eliot Lear <[email-masked]>
Date: Tue, 20 Apr 2010 19:14:21 +0200

> ... but note that the parenthetical question had to do with *EID* reachability, not locator reachability.
I'm going to hazard a guess that is what Dino meant, as locator 
reachability occurs as it does today.

---

From: John Scudder <[email-masked]>
Date: Wed, 21 Apr 2010 08:39:27 -0700

Eliot,

> reachability occurs as it does today.

I started to write a reply to this based on your guess that Dino actually meant "EID reachability" but quickly got mired down because you can't really take Dino's (quoted above) sentence and have it make sense if you s/locator/EID/g.  (For example, EIDs aren't part of locator-sets.)

So I guess I'll wait for Dino to clarify if he wants to.

--John

---

From: Dino Farinacci <[email-masked]>
Date: Wed, 21 Apr 2010 14:45:38 -0700

> Just replying to one point here.  Other replies as time allows.

Sure.

> I've only read the spec a few times so I may have missed where this  
> is spelled out.  Would you mind citing where this is stated in the  
> spec?

It is a bit implied in the following text but I will make it more  
clear in the -07 update.

    EID-to-RLOC Database:   a global distributed database that contains
       all known EID-prefix to RLOC mappings.  Each potential ETR
       typically contains a small piece of the database: the EID-to-RLOC
       mappings for the EID prefixes "behind" the router.  These map to
       one of the router's own, globally-visible, IP addresses.

> Is that "yes" to the "must be manually configured" question?  I  
> would have assumed so but you go on to talk about connecting to the  
> ALT and such, which would appear to be an unrelated (or tangentially  
> related) topic.

Yes, the database mapping must be consistently configured on all ETRs  
for a given LISP site.

Can you refer to my text that refers to the ALT? Thanks.

> reachability, not locator reachability.

If an ETR cannot connect to the site, it can set its LSB bit to 0.  
That is if all its site-facing interfaces go down, then it should do  
this. Other failure cases are more complicated but typically you won't  
find a site completely partitioning.

We have heard though that customers *may* want to deploy VRRP/HSRP  
between their xTRs at a site to detect site partitioning. We believe  
we don't need to add LISP mechanisms to solve this.

Sorry for the delayed response,
Dino

---

From: Vince Fuller <[email-masked]>
Date: Wed, 21 Apr 2010 15:22:24 -0700

> reachability, not locator reachability.

Can you please re-phrase the question and describe the context?

Except within the very limited scope of the ALT database mapping system, the
term "EID reachability" is something of an oxymoron.

Focusing only on use of LISP on the Internet (ignoring enterprise use cases),
an ETR may publish the set of RLOCs associated with an EID in the global
mapping database. It typically does so by registering that mapping with one
or more Map-Servers. In exceptional circumstances, an ETR may instead connect
to the ALT virtual overlay network and advertise an EID-prefix that covers
the EID using BGP on the ALT.

An ITR determines the RLOCs for a specific EID by querying the global
mapping system, typically by sending a Map-Request to a Map-Resolver and
awaiting a matching Map-Reply. Again, in exception circumstances, an ITR
may connect to the ALT and send a Map-Request for an EID on to the ALT which
then routes it toward the Map-Server or ETR that advertised the covering
EID prefix on to the ALT.

But this is all orthogonal to the issue of RLOC reachability (and is mostly
outside of the scope of draft-ietf-lisp anyway), so I'm not sure how to
interpret your comment/question above.

	Thanks,
	--Vince

---

From: John Scudder <[email-masked]>
Date: Wed, 21 Apr 2010 19:18:08 -0700

> Can you please re-phrase the question and describe the context?

This was covered in Comment 4 of the message Yakov sent to the list:

> term "EID reachability" is something of an oxymoron.

I guess the answer to comment 4 may shed some light on why it's an "oxymoron"?

> the EID using BGP on the ALT.

Understood.

> EID prefix on to the ALT.

Understood.

> interpret your comment/question above.

If it's still unclear, please let me know.

--John

---

From: John Zwiebel <[email-masked]>
Date: Wed, 21 Apr 2010 16:50:18 -1000

> This was covered in Comment 4 of the message Yakov sent to the list:

Why isn't there redundancy for the internal network?

Let me ask what would happen if a packet were to be received by
a CE router  running BGP under the same conditions?  If the CE router
advertises a more specific prefix, then the ETR1 and ETR2 can
also advertise a more specific prefix so the packet in question will
only go to the edge router with a route (reachability) to the EID in question.

So, assuming all edge routers (LISP or regular) advertise the same prefix,
no there isn't "some means" of fixing this problem using LISP -- yet.

---

From: John Scudder <[email-masked]>
Date: Thu, 22 Apr 2010 14:54:48 -0700

Hi Dino,

On Apr 21, 2010, at 5:45 PM, Dino Farinacci wrote:
[...]
> It is a bit implied in the following text but I will make it more  

Making it explicit would be a good start, thanks.  ("A bit implied" doesn't work very well for protocol specs, but you knew that.)

[...]
> Can you refer to my text that refers to the ALT? Thanks.

Sure:

"Yes, we recommend each ETR either is connected directly to the ALT or
                                     ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^
registers to a Map-Server, but if they do not they won't get Map-
Requests so they don't or can't send Map-Replies. We encourage all
ETRs to accept Map-Requests so the control-plane load can be balanced
across all the CPUs of the various ETRs."

> find a site completely partitioning.

(Just to be clear, "LSB bit" expands as "locator status bit bit", right?  Just making sure since the term isn't defined in the spec.)

If we look at 5.3 of the LISP spec, it sez:

     When a bit is set to 1, the
     ITR is indicating to the ETR the RLOC associated with the bit
     ordinal has up status.  See Section 6.3 for details on how an ITR
     can determine other ITRs at the site are reachable.  When a site
     has multiple EID-prefixes which result in multiple mappings (where
     each could have a different locator-set), the Locator Status Bits
     setting in an encapsulated packet MUST reflect the mapping for the
     EID-prefix that the inner-header source EID address matches.

The first sentence says the semantics of the LSB is that the RLOC is up, period.  As written, that would seem to indicate that the status given in the LSB should be applied to that RLOC (i.e., the thing indexed by that IP address), no matter which EID the RLOC is being used to reach.  It seems like you are suggesting that instead, the ITR has to track a different notion of "up status" for each (RLOC, EID) pair.  Is that right?  If so I think the spec should say so explicitly.

(By the way, there's also the question of what exactly the semantics of "up status" are.)

Anyway, let's proceed on the assumption that we're using your revised LSB semantics instead of what the spec says.  Let's take this with the aid of a few examples.  These all involve some variant of the site partitioning.

Example 1:
- EID1 is reached via ETR1 and ETR2, i.e., the mapping system thinks the locator-set for EID1 is {ETR1, ETR2}.
- At some particular time, ETR1 does not actually have connectivity to EID1 due to an intra-site problem.
- ITR1 wants to send traffic to EID1, so it performs a mapping lookup and gets back {ETR1, ETR2}.
- ITR1 picks ETR1 for whatever reason.  It sends an encapsulated data packet to ETR1's RLOC.
- ETR1 gets the packet.  Now what does it do?  The locator status bits which you suggest using would only be carried in return traffic since they are only present in the LISP data encapsulation header.  But since the forward traffic can't reach the destination host (remember ETR1 can't reach EID1) there is no return traffic.

Example 2:
- EID1 is reached via ETR1 and ETR2, i.e., the mapping system thinks the locator-set for EID1 is {ETR1, ETR2}.
- ETR1 initially can reach EID1.
- ITR1 sends traffic to EID1 via ETR1's RLOC.  The EID1 host sends return traffic.  For whatever reason, the return traffic uses ETR2 as its ITR.
- Now ETR1 loses connectivity to EID1.
- How does ETR1 tell ITR1?

Example 3: 
(this is a variant of example 1)
- EID1 is reached via ETR1 and ETR2, i.e., the mapping system thinks the locator-set for EID1 is {ETR1, ETR2}.
- At time T1 ITR1 wants to send traffic to EID1, so it performs a mapping lookup and gets back from ETR1 {ETR1, ETR2}. At this point ETR1 has connectivity to EID1.
- ITR1 picks ETR1 for whatever reason, and starts sending traffic to EID1.
- At time T2 there is no more traffic to send to EID1.
- At time T3 ETR1 loses connectivity to EID1 (due to intra-site problem)
- At time T4 ITR1 again wants to send traffic to EID1, and since ITR1 has the mapping, it sends traffic to ETR1.
- ETR1 gets the packet.  Now what does it do?  The locator status bits which you suggest using would only be carried in return traffic since they are only present in the LISP data encapsulation header. But since the forward traffic can't reach the destination host (remember ETR1can't reach EID1) there is no return traffic.

Example 4:
(another variant)
- EID1 is reached via ETR1 and ETR2, i.e., the mapping system thinks the locator-set for EID1 is {ETR1, ETR2}.
- At time T1 ITR1 wants to send TCP traffic to EID1, so it performs a mapping lookup and gets back from ETR1 {ETR1, ETR2}. Note that at this point ETR1 has connectivity to EID1. 
- ITR1 picks ETR1 for whatever reason, and starts sending traffic to EID1. Note that data in the TCP stream goes to EID1, and the only thing that EID1 sends back are TCP acks.
- At time T2 EID1 sends TCP ACK back to the source, and at time T3 ETR1 forwards this TCP ACK. Immediately after that ETR1 loses connectivity to EID1 (due to an intra-site problem).
- At time T4 ITR1 wants to send more data on the TCP session to EID1, and since ITR1 has the mapping, it sends traffic to ETR1.
- What happens with the TCP session ? (a) When TCP KeepAlives are in use, and (b) they aren't.

Of course there is also the variant where you've got a unidirectional stream from ITR1 to EID1.  It does happen.

Other interesting scenarios are possible but let's start with those.  

> Other failure cases are more complicated but typically you won't  

So is it the case that LISP isn't intended to cover the case where a site becomes partitioned?  If so this is a pretty important assumption and should be spelled out in the spec.  Maybe in Section 8?

> we don't need to add LISP mechanisms to solve this.

If they detect site partitioning, what would they do then?  (This is probably partly covered by the discussion above.)

--John

---

From: Dino Farinacci <[email-masked]>
Date: Thu, 22 Apr 2010 15:11:12 -0700

> Making it explicit would be a good start, thanks.  ("A bit implied"  
> doesn't work very well for protocol specs, but you knew that.)

You are right. You found a bug. I fixed it. I thank you.

> "Yes, we recommend each ETR either is connected directly to the ALT or

All I am trying to say is if they do not register a mapping with the  
mapping system, they will not get Map-Requests. So this ETR wouldn't  
*have to be* configured consistently with the other ETRs who *are*  
processing Map-Requests. But this is a degenerate case.

> (Just to be clear, "LSB bit" expands as "locator status bit bit",  
> right?  Just making sure since the term isn't defined in the spec.)

Yes.

> The first sentence says the semantics of the LSB is that the RLOC is  
> up, period.  As written, that would seem to indicate that the status  
> given in the LSB should be applied to that RLOC

Right, and to be clear here, "up" means the box is up and necessarily  
reachable from everywhere.

> (i.e., the thing indexed by that IP address), no matter which EID  
> the RLOC is being used to reach.  It seems like you are suggesting  
> that instead, the ITR has to track a different notion of "up status"  
> for each (RLOC, EID) pair.  Is that right?  If so I think the spec  
> should say so explicitly.

No. There is a one to many relationship. An EID-prefix maps to a  
locator-set. The reachability algorithms are used to decide which  
locator in the locator-set is reachable. There is no provision to say  
if the ETR the locator is assigned to can reach any EID in the EID- 
prefix.

> (By the way, there's also the question of what exactly the semantics  

Box is up and can receive control messages and get decap LISP packets.  
That is why we change it from loc-reach-bits to loc-status-bits.

> Anyway, let's proceed on the assumption that we're using your  
> revised LSB semantics instead of what the spec says.  Let's take  
> this with the aid of a few examples.  These all involve some variant  
> of the site partitioning.

No, let's not proceed.

> Example 1:
> - EID1 is reached via ETR1 and ETR2, i.e., the mapping system thinks  
> the locator-set for EID1 is {ETR1, ETR2}.
> - ETR1 gets the packet.  Now what does it do?  The locator status  
> bits which you suggest using would only be carried in return traffic  
> since they are only present in the LISP data encapsulation header.   
> But since the forward traffic can't reach the destination host  
> (remember ETR1 can't reach EID1) there is no return traffic.
>
> Example 2:
> - EID1 is reached via ETR1 and ETR2, i.e., the mapping system thinks  
> the locator-set for EID1 is {ETR1, ETR2}.
> - ITR1 sends traffic to EID1 via ETR1's RLOC.  The EID1 host sends  
> return traffic.  For whatever reason, the return traffic uses ETR2  
> as its ITR.
> - EID1 is reached via ETR1 and ETR2, i.e., the mapping system thinks  
> the locator-set for EID1 is {ETR1, ETR2}.
> - At time T1 ITR1 wants to send traffic to EID1, so it performs a  
> mapping lookup and gets back from ETR1 {ETR1, ETR2}. At this point  
> ETR1 has connectivity to EID1.
> - At time T4 ITR1 again wants to send traffic to EID1, and since  
> ITR1 has the mapping, it sends traffic to ETR1.
> - ETR1 gets the packet.  Now what does it do?  The locator status  
> bits which you suggest using would only be carried in return traffic  
> since they are only present in the LISP data encapsulation header.  
> But since the forward traffic can't reach the destination host  
> (remember ETR1can't reach EID1) there is no return traffic.
>
> Example 4:
> (another variant)
> - EID1 is reached via ETR1 and ETR2, i.e., the mapping system thinks  
> the locator-set for EID1 is {ETR1, ETR2}.
> - At time T1 ITR1 wants to send TCP traffic to EID1, so it performs  
> a mapping lookup and gets back from ETR1 {ETR1, ETR2}. Note that at  
> this point ETR1 has connectivity to EID1.
> - ITR1 picks ETR1 for whatever reason, and starts sending traffic to  
> EID1. Note that data in the TCP stream goes to EID1, and the only  
> thing that EID1 sends back are TCP acks.
> - At time T2 EID1 sends TCP ACK back to the source, and at time T3  
> ETR1 forwards this TCP ACK. Immediately after that ETR1 loses  
> connectivity to EID1 (due to an intra-site problem).
> - At time T4 ITR1 wants to send more data on the TCP session to  
> EID1, and since ITR1 has the mapping, it sends traffic to ETR1.
> Of course there is also the variant where you've got a  
> unidirectional stream from ITR1 to EID1.  It does happen.
>

All I want to say on this subject is that the EID reachability is no  
different than we have today. That is, today a router has a prefix, it  
does not indicate that every host in that prefix is reachable. So a  
packet will travel to a place in the topology where the packet will be  
dropped. Same thing when LISP is used.

The fact that the ITR could have selected another RLOC to get to the  
destination is something we are not going to solve. I'm sure you can  
see the complexity and machinery that would have to be implemented to  
do this.

> So is it the case that LISP isn't intended to cover the case where a  
> site becomes partitioned?  If so this is a pretty important  
> assumption and should be spelled out in the spec.  Maybe in Section 8?

It covers the case, but can route packets on the other side of the  
partition. You could have the ETR re-encapsulate to the other side of  
the partition, but we aren't going to spec that.

> If they detect site partitioning, what would they do then?  (This is  
> probably partly covered by the discussion above.)

Left up to the implementation.

Dino

---

From: Luigi Iannone <[email-masked]>
Date: Fri, 23 Apr 2010 09:52:28 +0200

Hi,

Versioning can help in those examples (see below).

But at one cost: issuing a new version number each time there is a status/reachability change and this is not always worth to do it.

Note that I am _not_ suggesting that the what I describe below should be added in the specs.
 
Rather I think that this just an example of operational use of the versioning feature. (alternatively, the examples below can be included in the versioning draft as example of use of the versioning feature). 

On Apr 22, 2010, at 23:54 , John Scudder wrote:

[snip]
> - ITR1 picks ETR1 for whatever reason.

You are assuming here that ITR1 get a mapping that states that ETR1 is able to reach EID1, but then the IGP goes down when the packets are actually sent. Right?

>  It sends an encapsulated data packet to ETR1's RLOC.
> - ETR1 gets the packet.  Now what does it do?  

If the destination site has updated the version number ETR1 will SMR ITR1 because the encapsulated packet will carry a stale version number. 

> The locator status bits which you suggest using would only be carried in return traffic since they are only present in the LISP data encapsulation header.  But since the forward traffic can't reach the destination host (remember ETR1 can't reach EID1) there is no return traffic.
> 
> Example 2:

Same solution like example 1, with the small difference that most probably is the traffic flowing from ETR2 to ITR1 that will notify ITR1 that it has a stale mapping.

> - ETR1 gets the packet.  Now what does it do?

Exactly the same case of example 1.

>  The locator status bits which you suggest using would only be carried in return traffic since they are only present in the LISP data encapsulation header. But since the forward traffic can't reach the destination host (remember ETR1can't reach EID1) there is no return traffic.
> 
> Example 4:
> (another variant)
> - What happens with the TCP session ?

Here you did not specify if EID1 is able to reach ETR2, because otherwise your TCP session is gone and there is nothing you can do (and this is not a problem specific to LISP)

Assuming that EID1 is able to reach ETR2 , using versioning as described above, you will have some packet losses (on which TCP will react as due to congestion) and then things will work again.

Luigi

> (a) When TCP KeepAlives are in use, and (b) they aren't.
>

---

From: John Scudder <[email-masked]>
Date: Fri, 23 Apr 2010 10:05:07 -0700

On Apr 22, 2010, at 6:11 PM, Dino Farinacci wrote:
[...]
> processing Map-Requests. But this is a degenerate case.

OK.  Although, even if they don't send Map-Replies wouldn't they have to be consistent with the other ETRs in order to produce correctly formatted locator status bits?  Which would still require them to be configured consistently?

> reachable from everywhere.

Did you mean "*not* necessarily"?

> if the ETR the locator is assigned to can reach any EID in the EID- 

I see.

>>> find a site completely partitioning.

was that if an ETR can't connect to *any* of the EID prefixes it serves, then it can set its LSB to zero.  Is that right?  BTW, the concept of 'site' isn't clearly defined in the spec, so I would prefer to discuss this in terms of EIDs instead of 'sites'.  I suggest you do too, or provide a clear definition of 'site'.

(Although as the examples you didn't want to discuss pointed out, there is a problem with this too, which is that the LSB is carried in return data traffic... which will not exist if there is no connectivity to any EID prefix, or in various other cases.)

> Box is up and can receive control messages and get decap LISP packets.  

This should be stated in the spec.  Also, shouldn't the full statement be more like 'Box is up and can receive and process control messages and LISP packets.'? Either way, if these are the semantics of the LSB (either what you wrote, or the revision I suggested), how should one xTR determine the status of another so that it can reflect it in the LSB?  In order for the locator status bits field to work, the various xTRs in a locator set do need to determine each other's status.  'Receive and process' is more challenging for a remote xTR to determine than 'is the remote xTR reachable' which is what I used to think the LSB meant.

> different than we have today.

I'm not interested (at this point) in comparing LISP to current routing.  I'm just trying to nail down what LISP's *own* characteristics are.  Only once that is done can there be meaningful comparison.  You may think you know exactly what LISP's characteristics are in this respect, but it's not evident from the spec, hence these questions.  We do seem to be making progress on answering the question of what LISP's characteristics are.

(As to the assertion above, suffice it to say that I think it's debatable, but I don't want to debate it right now since I think it's premature.)

> That is, today a router has a prefix, it  

Even if "we aren't going to spec that", at a minimum the spec ought to point out that this limitation exists and that solutions are considered out-of-scope.  Something like this:

"Section X.Y  Site Partition

"A site may be multihomed using two or more ETRs.  The hosts and infrastructure within a site will be addressed using one or more EID prefixes that are mapped to the RLOCs of the relevant ETRs in the mapping system.  One possible failure mode of such a deployment is for an ETR to lose reachability to one or more of the EID prefixes.  This architecture does not provide a solution for data to be reliably delivered to hosts within the site in this scenario; provision of such a solution is beyond the scope of this document."

--John

---

From: Darrel Lewis <[email-masked]>
Date: Fri, 23 Apr 2010 13:32:55 -0700

>   I suggest you do too, or provide a clear definition of 'site'.
> 

I agree and suggest that it would be good to add "LISP Site" to the Definition of Terms section of the document. 

How about:

LISP site:  A set of routers in an edge network that is under administrative
            control by a single organization.  LISP routers which reside in the
            edge network are the demarcation points to separate the edge network
            from the core network.

-Darrel

---

From: Scott Brim <[email-masked]>
Date: Fri, 23 Apr 2010 16:56:32 -0400

>             edge network are the demarcation points to separate the edge network

Why under control of a single organization?  All that's needed is that
they act in concert.  I would just require the minimum.

Scott

---

From: Dino Farinacci <[email-masked]>
Date: Fri, 23 Apr 2010 15:11:46 -0700

>> I agree and suggest that it would be good to add "LISP Site" to the  
>> Definition of Terms section of the document.
>>
>> How about:
>>

Because that is how an IGP based AS is defined to today. We are not  
changing that.

Dino

---

From: Scott Brim <[email-masked]>
Date: Fri, 23 Apr 2010 20:31:45 -0400

> Because that is how an IGP based AS is defined to today. We are not

OK so the issue is they need to be connected by an IGP.

---

From: Dino Farinacci <[email-masked]>
Date: Fri, 23 Apr 2010 19:05:55 -0700

> OK.  Although, even if they don't send Map-Replies wouldn't they  
> have to be consistent with the other ETRs in order to produce  
> correctly formatted locator status bits?  Which would still require  
> them to be configured consistently?

Yes, if they are also ITRs and set the L-bit to 1 in data packets.

> Did you mean "*not* necessarily"?

Yes, typo. Sorry.

> was that if an ETR can't connect to *any* of the EID prefixes it  
> serves, then it can set its LSB to zero.  Is that right?  BTW, the  
> concept of 'site' isn't clearly defined in the spec, so I would  
> prefer to discuss this in terms of EIDs instead of 'sites'.  I  
> suggest you do too, or provide a clear definition of 'site'.

Right, we put a definition in the -07 Definition of Terms section.

> (Although as the examples you didn't want to discuss pointed out,  
> there is a problem with this too, which is that the LSB is carried  
> in return data traffic... which will not exist if there is no  
> connectivity to any EID prefix, or in various other cases.)

Let me remind you that we say the LSBs are hints. There are specific  
applications where they are useful.

> This should be stated in the spec.  Also, shouldn't the full  
> statement be more like 'Box is up and can receive and process  
> control messages and LISP packets.'? Either way, if these are the  
> semantics of the LSB (either what you wrote, or the revision I  
> suggested), how should one xTR determine the status of another so  
> that it can reflect it in the LSB?  In order for the locator

 From the IGP.

> status bits field to work, the various xTRs in a locator set do need  
> to determine each other's status.  'Receive and process' is more  
> challenging for a remote xTR to determine than 'is the remote xTR  
> reachable' which is what I used to think the LSB meant.
>
> I'm not interested (at this point) in comparing LISP to current  
> routing.  I'm just trying to nail down what LISP's *own*  
> characteristics are.  Only once that is done can there be meaningful  
> comparison.  You may think you know exactly what LISP's  
> characteristics are in this respect, but it's not evident from the  
> spec, hence these questions.  We do seem to be making progress on  
> answering the question of what LISP's characteristics are.

Sure, understand. We have been improving the spec over the last 3  
years and continue to doing so.

> (As to the assertion above, suffice it to say that I think it's  
> debatable, but I don't want to debate it right now since I think  
> it's premature.)

Fair enough.

> Even if "we aren't going to spec that", at a minimum the spec ought  
> to point out that this limitation exists and that solutions are  
> considered out-of-scope.  Something like this:
>
> "A site may be multihomed using two or more ETRs.  The hosts and  
> infrastructure within a site will be addressed using one or more EID  
> prefixes that are mapped to the RLOCs of the relevant ETRs in the  
> mapping system.  One possible failure mode of such a deployment is  
> for an ETR to lose reachability to one or more of the EID prefixes.   
> This architecture does not provide a solution for data to be  
> reliably delivered to hosts within the site in this scenario;  
> provision of such a solution is beyond the scope of this document."

Noted. We will include this in the -08 revision.

Thanks again,
Dino

---

From: John Scudder <[email-masked]>
Date: Mon, 26 Apr 2010 07:57:08 -0700

> I agree and suggest that it would be good to add "LISP Site" to the Definition of Terms section of the document. 

It might be helpful to provide examples of what a "site" can and can't be, e.g. it seems pretty clear a site can be a single physical office (and associated network infrastructure, etc).  Can a site be:

- A single home?
- A single host which speaks LISP natively?
- A global enterprise with extensive internal infrastructure?

Thanks,

--John