Subject: [lisp] AD review of draft-ietf-lisp-alt

From: Jari Arkko <[email-masked]>
Date: Thu, 08 Sep 2011 01:11:54 +0300

I have reviewed this draft. Like the other drafts that I have read, it was well written and pleasure to read. Thanks for doing good work.

This draft does have some hand waving in some aspects (like exactly how the ALT is built), but that's acceptable for an experimental specification. I did have four remaining issues that I would like to talk about though. One is a question and for the other three I have some suggested way forward. I also have a small set of editorial suggestions at the end of this mail. Please respond and/or revise draft so that we can send this one to IETF Last Call.

The question is about the treatment of RLOC source and EID destination addresses when something bad happens in ALT (no route or some other problem, say MTU limitation). Do you expect ICMP messages be sent back, and if so, how?

The use of RLOC source addresses seems to have an effect on intermixing IPv4 and IPv6 addresses. It would perhaps be desirable in some cases to allow IPv4 RLOCs while working with IPv6 EIDs, for instance. But those RLOCs cannot be directly used on IPv6 packets that carry IPv6 destination addresses. (At least not without agreeing that some form of mapped addresses are used. But that doesn't work for carrying an IPv6 source address in an IPv4 packet.) My suggestion is to add some text about this.

As this is an experimental specification and because we do not fully understand all aspects of its operation, I think it is useful to add some description at the beginning about the areas where further experience is needed.

Finally, I'm concerned about the Section 6.1 recommendation to use a "new numbering space that is unrelated to the ASNs used in the global routing system". First of all, its not clear to me what you are saying here. Are you (a) recommending that the 32-bit ASN space can just be reused and it doesn't matter if ALT uses the same numbers as someone in the current Internet? Or are you saying (b) that some new allocations (or perhaps even a small subspace) should be allocated from the existing ASN space and used for ALT, without colliding with any existing allocations? I think the former would be problematic, because its hard to prevent accidental leakage. But getting some new numbers for ALT usage would perfectly fine, IMO. I'm also wondering if we should just allocate a new SAFI (even if we do not require that legacy router implementations use it).

Here's some suggested text changes for Section 6.1:

OLD:
    To avoid confusion, it needs to be stressed that that
    these LISP+ALT ASNs use a new numbering space that is unrelated to
    the ASNs used by the global routing system.
NEW:
    To avoid confusion, it needs to be stressed that that
    these LISP+ALT ASNs should use newly allocated ASN numbers that are unrelated to
    the ASNs used by the global routing system.

And some suggested new text for Section 1, to be inserted just before the last paragraph:

NEW:
   This specification is experimental, and there are areas where further experience is needed
   in order to understand the best implementation strategies, operational models, and impacts to
   the rest of the Internet. These areas include:

   - application effects of on-demand route map discovery
   - the impacts of using map requests versus carrying initial packets in ALT
   - the best practical ways to build ALT hierarchies
   - effects of route leakage from ALT to the current Internet
   - effects of exceptional situations, such as denial-of-service attacks

   Experimentation, measurements, and deployment experience on these aspects is
   appreciated. While the theoretical impacts are often understood (such as an ALT
   lookup causing a potential delay for the first packet destined to a given network),
   it is less clear what effects, if any, these may have with real world traffic patterns.

   ALT cannot support mixed use of IPv4 and IPv6 addresses, as it requires the use of the same
   address family for both RLOCs and EIDs.

(Feel free to suggest alternate text.)

Editorial:

> ... little, if any, LISP-specific modifications should be
> needed for such devices to be deployed on the ALT.

Perhaps you could add a reference to Section 4.2 that discusses what is possible and is not possible with pure off-the-shelf routers. Otherwise the statement feels a little bit fuzzy.

s/on the ALT./on the ALT (see Section 4.2)./

>    == Outdated reference: A later version (-15) exists of draft-ietf-lisp-14
>
>    == Outdated reference: A later version (-11) exists of draft-ietf-lisp-ms-09
>

(from idnits, please fix)

>     Endpoint ID (EID):  A 32-bit (for IPv4) or 128-bit (for ipv6) value
>     reachability is carried as IPv4 or ipv6 NLRI without modification
>

s/ipv6/IPv6/

> This might be done minimize packet loss and to
>        probe for the mapping.

... might be done to minimize ...

>     low performance expected of a tuneled topology, ALT Routers (and Map
>

s/tuneled/tunneled/

Jari

---

From: Vince Fuller <[email-masked]>
Date: Thu, 8 Sep 2011 13:13:36 -0700

Hi Jari-

Thanks for the review, comments, and suggestions. Here are my thoughts
in response.

> The question is about the treatment of RLOC source and EID destination 
> addresses when something bad happens in ALT (no route or some other 
> problem, say MTU limitation). Do you expect ICMP messages be sent back, and 
> if so, how?

An ALT node can attempt to return an ICMP message in response to an error
condition (next hop unreachable, etc.) but there is obviously no guarantee
that it will be received since the RLOC destination may not be reachable
by a device which can only route on the ALT EID numbering space. Such a
failure will result in the sender of the Map-Request (and MR or ETR) giving
up after several retransmissions and treating the destination as unreachable
via LISP. Depending on how the source is configured, it may then try to use
LISP Interworking mechanisms to route "natively" to the destination.

> The use of RLOC source addresses seems to have an effect on intermixing 
> IPv4 and IPv6 addresses. It would perhaps be desirable in some cases to 
> allow IPv4 RLOCs while working with IPv6 EIDs, for instance. But those 
> RLOCs cannot be directly used on IPv6 packets that carry IPv6 destination 
> addresses. (At least not without agreeing that some form of mapped 
> addresses are used. But that doesn't work for carrying an IPv6 source 
> address in an IPv4 packet.) My suggestion is to add some text about this.
> 
> As this is an experimental specification and because we do not fully 
> understand all aspects of its operation, I think it is useful to add some 
> description at the beginning about the areas where further experience is 
> needed.

This situation is supposed to be covered by use of the ITR RLOC field. Use
of an ITR-provided RLOC is mentioned in section 5 and is further described
in [LISP] section 6.1.2 (where the field is shown in the Map-Request
message format) and section 6.2 (Routing Locator Section).

I will add appropriate references to [LISP] regarding ITR RLOC usage and
will explicitly mention the mixed-AF use case. Will that resolve this issue?

Keep in mind that the ALT exists for one purpose: so that a Map-Request for
an EID can be forwarded toward the ETR that can provide a Map-Reply. Traffic
is not bi-directional and the ITR RLOC information is included in the
Map-Request so that a responding ETR or MS can return a Map-Reply directly
to the Map-Request originator, bypassing the ALT.

> Finally, I'm concerned about the Section 6.1 recommendation to use a "new 
> numbering space that is unrelated to the ASNs used in the global routing 
> system". First of all, its not clear to me what you are saying here. Are 
> you (a) recommending that the 32-bit ASN space can just be reused and it 
> doesn't matter if ALT uses the same numbers as someone in the current 
> Internet? Or are you saying (b) that some new allocations (or perhaps even 
> a small subspace) should be allocated from the existing ASN space and used 
> for ALT, without colliding with any existing allocations? I think the 
> former would be problematic, because its hard to prevent accidental 
> leakage. But getting some new numbers for ALT usage would perfectly fine, 
> IMO. I'm also wondering if we should just allocate a new SAFI (even if we 
> do not require that legacy router implementations use it).
> 

Thanks for the suggested text; I'll add to draft-ietf-lisp-alt-08.

>   Experimentation, measurements, and deployment experience on these aspects 
>   is appreciated. While the theoretical impacts are often understood (such
>   as an ALT lookup causing a potential delay for the first packet destined
>   to a given network), it is less clear what effects, if any, these may
>   have with real world traffic patterns.
> 

Again, thanks for the suggestion. I'll start with this and re-work it a bit
for incorporation into section 1 of the -08 draft.

> Perhaps you could add a reference to Section 4.2 that discusses what is 
> possible and is not possible with pure off-the-shelf routers. Otherwise the 
> statement feels a little bit fuzzy.
> 

Thanks, will fix in -08.

> (from idnits, please fix)

Thanks, will fix in -08.

Of course, creating -08 will result in an idnit for those documents, since
they reference ALT-08; will plan to fix those as part of their AD reviews.

> ... might be done to minimize ...

Thanks, will fix in -08.

> >    low performance expected of a tuneled topology, ALT Routers (and Map

Thanks, will fix in -08.

	--Vince

---

From: Jari Arkko <[email-masked]>
Date: Thu, 08 Sep 2011 23:34:04 +0300

Vince,

Thanks for your quick response. I think you solved my mixed case issue.

I'd still like to understand the ICMP message situation better. I realize that the traffic is supposed to be unidirectional, but how would, say, lower MTU somewhere along the path be dealt with? I'm not necessarily asking for a change in what the spec does, but it should be clear about the consequences.

Jari

---

From: Dino Farinacci <[email-masked]>
Date: Thu, 8 Sep 2011 13:37:41 -0700

> I'd still like to understand the ICMP message situation better. I realize that the traffic is supposed to be unidirectional, but how would, say, lower MTU somewhere along the path be dealt with? I'…

Since Map-Requets are small, it is unlikely that an MTU violation will occur for a Map-Request forwarded on the ALT. But if the case did arise, there would be fragmentation and reassembly at each GRE end-point since a Map-Request, when sent from one ALT node to another, is encapsulated in a GRE tunnel header.

Dino

> 
> Jari
> 
> _______________________________________________
> lisp mailing list
> [email-masked]
> https://www.ietf.org/mailman/listinfo/lisp

---

From: Vince Fuller <[email-masked]>
Date: Thu, 8 Sep 2011 13:47:37 -0700

> I'd still like to understand the ICMP message situation better. I realize 
> that the traffic is supposed to be unidirectional, but how would, say, 
> lower MTU somewhere along the path be dealt with? I'm not necessarily 
> asking for a change in what the spec does, but it should be clear about the 
> consequences.

Under normal circumstances, the only thing that should be send across the
ALT would be a Map Request, which should not cause MTU problems. Of course,
if a (deprecated) Data Probe were used, then all bets are off.

If a Map Request or Data Probe were to hit an MTU problem or otherwise be
undeliverable across the ALT and no ICMP message could be returned from the
ALT node where the failure occurred to the Map Request originator, then
the originator would retry sending the Map Request several times before
timing-out. If such a timeout occurred, then the destination would be
considered not LISP-capable and LISP encapsulation would not be done
toward it.

	--Vince

---

From: Jari Arkko <[email-masked]>
Date: Tue, 13 Sep 2011 23:45:23 +0300

Vince, Dino,

> Since Map-Requets are small, it is unlikely that an MTU violation will occur for a Map-Request forwarded on the ALT. But if the case did arise, there would be fragmentation and reassembly at each G…

I'm looking at the Map-Request message format and cannot quite convince myself that there would never be a situation where MTU would be exceeded. 99.999% of the time it would be fine, but I think we need to describe (if not fix) what happens when its not. How about adding the following item to the Section 1 experimental list:

"o  effects of limited possibilities for returning an ICMP message from ALT"

and this text somewhere:

"Given that packets delivered through ALT have an RLOC source address and an EID destination address, returning an ICMP error may not always be possible. Since Map-Request messages are typically small, it is unlikely that an MTU violation will occur for a Map-Request forwarded on the ALT and if it did, it may be possible to deal with it through fragmentation of GRE packets. But in the unlikely event of having to generate an ICMP error due to MTU or other problems, an ALT node MAY attempt to deliver the error to the source RLOC outside ALT. If it turns out to be not possible to deliver the ICMP error, the sender of the Map-Request message will retry, and failing again, will time out.This could lead to considering the dstination not LISP-capable. Note that ICMP errors are more likely with Data Probes, and may lead to the unreported loss of the encapsulated data packet."

Or something along those lines.

Jari

---

From: Vince Fuller <[email-masked]>
Date: Wed, 14 Sep 2011 22:18:47 -0700

> I'm looking at the Map-Request message format and cannot quite convince 
> myself that there would never be a situation where MTU would be exceeded. 
> 99.999% of the time it would be fine, but I think we need to describe (if 
> not fix) what happens when its not. How about adding the following item to 
> the Section 1 experimental list:

I don't really understand how MTU could be an issue here. If there is an
MTU restriction, fragmentation would occur upon entry into an ALT GRE
tunnel and reassembly would be done when the ALT datagram is received by
an ETR.

> "o  effects of limited possibilities for returning an ICMP message from ALT"

It is certainly possible for an intermediate ALT router to be unable to
forward an ALT Datagram for some other reason, such as there being no
matching next hop (i.e. the destination is not an EID and therefore is not
reachable via the ALT). In such cases, the ALT router which cannot forward
the datagram would drop it. Section 4.2 describes what an ETR can do if
it receives an ALT datagram destined for a non-EID: it returns a negative
Map-Reply. An ALT router that understands LISP (i.e. a Map Server or Map
Resolver connected to ALT) could similarly send a negative Map-Reply if
it cannot forward a Map-Request over the ALT.

An ALT router could attempt to return an ICMP message to the RLOC source
in a Map-Request received over the ALT. Whether that ICMP message would
be received by the Map-Request source (ITR) would depend on whether the
ALT router has legacy Internet connectivity in addition to being on the
ALT virtual network. The current Cisco LISP implementations do not attempt
to return ICMP messages when a failure occurs forwarding on the ALT VRF.

If an ITR were to send a Map-Request into the ALT that cannot be delivered
over the ALT due to some IP-layer problem, it may very well not receive
any positive failure indication. Since such a scenario could only arise
through misconfiguration or another serious problem on the ALT network,
having an ITR treat the destination as non-LISP-reachable is, in my opinion,
the right thing to do, so I don't see a need for additional machinery.

> and this text somewhere:
> 
> "Given that packets delivered through ALT have an RLOC source address and 
> an EID destination address, returning an ICMP error may not always be 
> possible. Since Map-Request messages are typically small, it is unlikely 
> that an MTU violation will occur for a Map-Request forwarded on the ALT and 
> if it did, it may be possible to deal with it through fragmentation of GRE 
> packets. But in the unlikely event of having to generate an ICMP error due 
> to MTU or other problems, an ALT node MAY attempt to deliver the error to 
> the source RLOC outside ALT. If it turns out to be not possible to deliver 
> the ICMP error, the sender of the Map-Request message will retry, and 
> failing again, will time out.This could lead to considering the dstination 
> not LISP-capable. Note that ICMP errors are more likely with Data Probes, 
> and may lead to the unreported loss of the encapsulated data packet."

If you feel it is important enough, we can certainly add text of this sort
stating that an ALT router may attempt to return an ICMP message if it cannot
forward an ALT Datagram. None of the authors think that this is really
necessary.

	--Vince

---

From: Jari Arkko <[email-masked]>
Date: Mon, 19 Sep 2011 14:39:16 +0300

Vince,

> tunnel and reassembly would be done when the ALT datagram is received by

OK.

(Maybe this should be said somewhere in the draft. Or maybe it already is and I just missed it...)

> reachable via the ALT).

OK. Lets focus on the general issue, not the specific MTU problem.

> In such cases, the ALT router which cannot forward

Yes, and this is good. But FWIW -- and maybe this is one reason why the issue has been unclear to me -- Section 4.2 seemed to talk about this only in the context of aggregate prefixes and their holes. And it does talk about only ETRs, not ALT routers. Would it be possible to add something from your above description to the text?

> the right thing to do, so I don't see a need for additional machinery.

This may well be so. And I agree that additional machinery might not be helpful.

> forward an ALT Datagram. None of the authors think that this is really

I think I understand the situation now better, thanks for the explanation. I don't think we need to specify a way to send ICMP messages, but I still think it would be reasonable to explain the situation to the reader. Here's a second try for new text:

"Given that packets delivered through ALT have an RLOC source address and an EID destination address, returning an ICMP error may not be possible. As described in Section 4.2., the reception of a packet that has no destination in ALT can be treated by ETRs or even the ALT routers by sending a negative Map-Reply. It is expected that MTU problems are handled as a part of GRE tunnels. Other problems can only occur through misconfiguration or other serious problems with the ALT network. The ALT design treats the remaining cases as packet loss. The sender of a Map-Request may retry sending a message, and failing again, will eventually time out. This leads to treating the destination as not reachable through LISP, which is is the the best or only course of action that can be performed under such circumstances."

Comments? Feel free to propose your own text. But I'd like to have something in the draft about this, and this is the only thing preventing the draft going to Last Call...

Jari

---

From: Vince Fuller <[email-masked]>
Date: Mon, 19 Sep 2011 15:45:17 -0700

> Yes, and this is good. But FWIW -- and maybe this is one reason why the 
> issue has been unclear to me -- Section 4.2 seemed to talk about this only 
> in the context of aggregate prefixes and their holes. And it does talk 
> about only ETRs, not ALT routers. Would it be possible to add something 
> from your above description to the text?

It turns out that there was a placeholder for a section 5.3 that was
intended to describe special processing by intermediate ALT routers. Here's
what I propose to put there to describe forwarding failures:

| 5.3.  ALT Datagram forwarding falure
| 
|    Intermediate ALT routers, forward ALT Datagrams using normal, hop-by-
|    hop routing on the ALT overlay network.  Should an ALT router not be
|    able to forward an ALT Datagram, whether due to an unreachable next-
|    hop, TTL exceeded, or other problem, it has several choices:
| 
|    o  If the ALT router understands the LISP protocol, as is the case
|       for a Map Resolver or Map Server, it may respond to a forwarding
|       failure by returning a negative Map-Reply, as described in
|       Section 4.2 and [LISP-MS].
| 
|    o  If the ALT router does not understand LISP, it may attempt to
|       return an ICMP message to the source IP address of the packet that
|       cannot be forwarded.  Since the source address is an RLOC, an ALT
|       router can only do this if it has "native" Internet connectivity
|       in addition to its ALT overlay network connectivity.
| 
|    o  A non-LISP-capable ALT router that does not have "native" Internet
|       connectivity will drop the packet.
| 
|    [LISP] and [LISP-MS] define how the source of an ALT Datagram should
|    handle each of these cases.  The last case, where an ALT Datagram is
|    silently discarded, will generally result in several retransmissions
|    by the source, followed by treating the destination as unreachable
|    via LISP when no Map-Reply is received.  If a problem on the ALT is
|    severe enough to prevent ALT Datagrams from being delivered to a
|    specific EID, this is probably the only sensible way to handle this
|    case.
| 
|    Note that the use of GRE tunnels should prevent MTU problems from
|    ever occurring on the ALT; an ALT Datagram that exceeds an
|    intermediate MTU will be fragmented at that point and will be
|    reassembled by the target of the GRE tunnel.

Does this address your concerns?

	--Vince

---

From: Vince Fuller <[email-masked]>
Date: Mon, 19 Sep 2011 16:34:05 -0700

As Dino pointed out in a private message to me, the fact that the ALT network
is an overlay on the Internet means that it is unlikely that an ALT Router
would not have "native" connectivity. Here's a slightly simplified section
5.3 that eliminates the reference to such an unlikely situation.

	--Vince

		--------------------

5.3.  ALT Datagram forwarding falure

   Intermediate ALT Routers, forward ALT Datagrams using normal, hop-by-
   hop routing on the ALT overlay network.  Should an ALT router not be
   able to forward an ALT Datagram, whether due to an unreachable next-
   hop, TTL exceeded, or other problem, it has several choices:

   o  If the ALT Router understands the LISP protocol, as is the case
      for a Map Resolver or Map Server, it may respond to a forwarding
      failure by returning a negative Map-Reply, as described in
      Section 4.2 and [LISP-MS].

   o  If the ALT Router does not understand LISP, it may attempt to
      return an ICMP message to the source IP address of the packet that
      cannot be forwarded.  Since the source address is an RLOC, an ALT
      Router would send this ICMP message using "native" Internet
      connectivity, not via the ALT overlay.

   o  A non-LISP-capable ALT Router may also choose to silently drop the
      non-forwardable ALT Datagram.

   [LISP] and [LISP-MS] define how the source of an ALT Datagram should
   handle each of these cases.  The last case, where an ALT Datagram is
   silently discarded, will generally result in several retransmissions
   by the source, followed by treating the destination as unreachable
   via LISP when no Map-Reply is received.  If a problem on the ALT is
   severe enough to prevent ALT Datagrams from being delivered to a
   specific EID, this is probably the only sensible way to handle this
   case.

   Note that the use of GRE tunnels should prevent MTU problems from
   ever occurring on the ALT; an ALT Datagram that exceeds an
   intermediate MTU will be fragmented at that point and will be
   reassembled by the target of the GRE tunnel.

---

From: Jari Arkko <[email-masked]>
Date: Tue, 20 Sep 2011 07:56:15 +0300

Yes, this text is great. Thank you. Put it in the draft and lets move forward.

Jari

---

From: Jari Arkko <[email-masked]>
Date: Tue, 20 Sep 2011 07:57:36 +0300

This makes sense.

Jari