Subject: Re: [lisp] RtgDir review: draft-ietf-lisp-alt-03.txt

From: Vince Fuller <[email-masked]>
Date: Mon, 8 Mar 2010 11:08:21 -0800

Hi John-

Thanks for the thorough review and comments. I have incorporated most of
them in to the latest draft, which is attached along with diffs from -02.

To respond specifically to several several of your points:

> The document's goal is to "[describe] a method of building an
> Alternative Logical Topology" I'm not sure that I'd say it achieves
> this goal.  Some key aspects of building such a topology are how to
> interconnect the member routers and how to perform aggregation at
> the various levels; these are addressed in general terms but not in
> sufficient detail to tell someone how do it for the ALT.  Also
> missing is any information of how one might connect to the deployed
> ALT although this may be a non-goal, since it's beyond the scope of
> just "a method of building".  I would buy "outline an architecture
> for building" as a description of the document as it stands.  If the
> document actually wants to do what the abstract says, I think it
> needs more detail especially in sections 7-9, preferably with
> examples.

This document is intended to describe, in general terms, how to build the
ALT, using BGP and GRE, and what the different components of the ALT are.
A companion deployment guide document, which will cover more specific
operational aspects of the ALT, is already in the works.

> The document doesn't attempt to tackle questions of deployment at
> scale.  Some of the open technical issues are identified (e.g. in
> the security section) although none of the operational ones are
> discussed.  Examples of potential operational issues might be the
> manageability of a large-scale tunneled overlay network, the ability
> of operators to troubleshoot customer issues when there is no easy
> way to distinguish an "EID" from a conventional IP address, and how
> to handle sites that leak their "EID" prefix into the regular
> Internet routing, either accidentally or on purpose.

See above regarding the deployment guide. The issue of distinguishing an
EID from a "conventional IP address" is a difficult one that is inherent
in trying to build a id/locator separation system that uses "addresses"
that are backward-compatible with existing host usage. 

> The Security section discusses a number of possible attacks.  Here's
> another one: "EID" prefix injection into the plain old Internet
> routing.  Such an attack would appear to be at increased likelihood
> of success compared to an identical attack against a normal IP
> prefix.  This is because per [LISP] only a highly aggregated "EID"
> prefix would be advertised into the plain old Internet, making a
> longer-prefix attack likely to succeed.  This is admittedly really a
> shortcoming in the BGP "security" model and would be addressed by
> any solution which addresses the underlying problem; still, the
> problem seems to be exacerbated by LISP.  Presumably such an attack
> wouldn't be effective to divert traffic from a LISP source, but it
> would work against non-LISP sources (i.e. the majority of the
> Internet).  This may be a case of "can't make an omelette without
> breaking eggs" but it's still worth noting.

Added to the Security Considerations section.

> The document emphasizes that the ALT is to be used only to forward
> ALT datagrams.  Since "ALT datagrams" are just IP packets as far as
> the ALT Routers are concerned, are technical means needed to enforce
> this assumption?  This would seem to deserve treatment in the main
> body of the document, or at least in the Security section.  As it
> is, it would appear easy to inject arbitrary packets into the ALT.

Discussion added about rate-limiting of and possible filtering of non-ALT-
Datagrams (Map Reqeusts and Data Probes) added as appropriate.

> S5.3 claims "LISP+ALT ... reduces ... OPEX".  I don't think this
> claim is justified by evidence.  (I can see a stronger case for
> claiming it reduces vendor development costs.)  I understand that
> there is goodness from not having to train folks in the use of a new
> protocol.  On the other hand, much will depend on how the overlay
> network is constructed and maintained.  Until experience is gained
> in that, it's not knowable whether opex is high or low overall.
> Furthermore, the statement begs the question, reduces opex compared
> to what?  For these reasons, I think it would be better to remove
> this claim.

Section rewritten with clarifying text added.

> ALT's claims to scalability rest on high levels of aggregation with
> no hole punching.  This has implications on the necessary business
> relationships between EID users and EID suppliers, and the types of
> suballocation that can and cannot be done.  Most importantly, it
> implies that EID users will be locked in to their EID supplier, of
> course unless they are willing to renumber into different EID space
> -- but freedom from renumbering is a goal of LISP.  This consequence
> probably should be spelled out in some fashion.  This might come
> naturally from a much more detailed description of specifics as I
> raise in other points, or it might need its own section.

This is partly addressed (no pun intended) by changes to the aggregation
example, which now specifically discusses how holes are handled. There is
also new clarifying text that emphasizes that the ALT is a tunneled
topology, so re-homing attachment points to facilitate aggregation should
be relatively simple. A more thorough discussion of business relationships
is, IMHO, beyond the scope of a technical IETF document and it a matter
for further study as EID/RLOC separation technology evolves.

> S.1 The document cites RFC 2119 but barely uses the terminology.
> It's limited to one MUST in S.3 and one more in S.4.  There are many
> places where 2119 type language would clarify things.  For example,
> look at the uses of "should".  A number could be turned into MUST or
> in some cases, SHOULD.  (Others really are the plain old English
> "should".)  Although one option would be to ditch the use of 2119
> language altogether (including removing the citation) I'd prefer to
> see the document worked over to use it properly.  To take just one
> example, S6.1 ("Changes to ITR behavior with LISP+ALT") seems to cry
> out for some MUSTs where "should" is used.

References to RFC 2119 and use of terminology have been removed.

> S.2 notes that "an important design goal of LISP+ALT is to minimize
> the number of changes to existing hardware and/or software".  This
> begs the question of what the other design goals were.  It would be
> useful to know this, as a way of letting the reader evaluate how
> well LISP+ALT meets those goals.  Were there any explicit non-goals?

Section 2 rewritten to address this.

> S.3 says that an EID is the address used in the first "LISP header"
> of a packet.  Does the LISP architecture refer to the header of the
> encapsulated packet as a "LISP header"?  If so I think that is
> unfortunate; it's really just an IP header (and that's how the host
> that emitted it thinks of it; isn't that the point?).  But, if so
> then I think there needs to be a discussion of this renaming of the
> IP header and just exactly what "LISP header" means.  OTOH, maybe
> this is just an editorial error, in which case it looks as though it
> should say "IP header".  (Based on the definition of "LISP header"
> in [LISP] I think it's just an editorial error.)  Also, this
> paragraph says "system" where I think it should say "host".

The term "LISP header" is defined in the LISP document. It is slightly
more than just an encapsulating IP header as it also includes a UDP
header (required for LAG traffic distribution according to those with
operational networks who have worked with the authors) with destination
port 4141 (LISP-encapsulated user data) or 4142 (LISP control). For
LISP control messages, there are additional defined fields.

> S.3, the definition of EID Prefix Reachability refers to "the ETR
> (or its proxy)".  The notion of an ETR's proxy isn't amplified on in
> the spec.  Either do so, or cite another document which does so
> ([LISP-IW]?).

The "or its proxy" is an obsolete term from an early version of the
document and has been removed. Thanks for catching this.

> Section 4.3 notes that "Data Probes ... should be considered
> experimental."  Isn't all of LISP experimental, by definition?  When
> seen in that light, this statement is quite odd.  I'm not really
> sure what your goal is with this phrasing so it's hard for me to
> suggest different wording, but something needs to change.  (This
> would also seem to be a good place for 2119 type language,
> e.g. "MUST be disabled by default".)  It might help clarity to
> restructure the doc making Map-Request the only lookup method
> documented in the main doc, and move all discussion of Data Probes
> to an appendix.

This section has been reworded to more clearly describe when Data Probes
may be used, which is basically only when doing specific experiments
intended to demonstrate whether they are useful. We expect that such
experiments will be completed before LISP and LISP+ALT move towards
standards adoption (if that occurs) and that revised versions of the
specs will either eliminate Data Probes completely or will provide a
more thorough description of how and when it is appropriate to use them.

> S5.1, second paragraph, "additional packets destined to the given
> EID prefix are routed directly to a viable ETR without use of the
> ALT, until either the entry's TTL has expired, or the ITR can
> otherwise find no reachable ETR."  Considering that the next
> sentence says that non-reachability of all the RLOCs in the mapping
> does not constitute "can otherwise find no reachable ETR" I wonder
> what WOULD be covered by this clause?

This tortured text was attempting to describe negatve cache entries and
dates back to an early version of the document before we'd formalized
what that meant. I have rewritten it for clarity.

> S5.2, third paragraph, provides an example of aggregation that is
> mostly holes.  Either the paragraph should be updated to get rid of
> the holes (replace /24 with /18) or the aggregation-across-holes
> should be explicitly addressed.

Rewritten and expanded to explicitly describe how holes works.

> S5.3, it took me several reads to understand what point "[a]lso,
> since tunnel IP addresses are local in scope, no coordination is
> needed for their assignment" is getting at.  Maybe something like
> "... since any given tunnel in the ALT is only relevant to the pair
> of routers that connects over it, the only requirement of the IP
> addresses used to establish that tunnel is that the involved routers
> should be able to reach each other."  Or something like that.

Rewritten pretty much as you suggested.

> Section 7.2 (SAFI for LISP+ALT) asks whether it would be prudent to
> use a different SAFI to syntactically distinguish ALT routes from
> Internet routes.  This seems like a good idea to me since it would
> help eliminate one hard-to-debug routing problem (and source of
> increased opex, see other comments).  However, if this is done it
> has the disadvantage of eliminating the possibility of using COTS
> routers for ALT routers.

Rewitten to point out this trade-off.

> S8.1 "First, since reachability of RLOCs is learned through the LISP
> ITR-ETR exchange, "flapping" (frequent BGP updates and withdrawals)
> is not likely, and mapping information cannot become "stale" due to
> slow propagation through the ALT BGP mesh."  I can't tell what this
> sentence is trying to say.

Rewritten and expanded to better explain this.

> S8.1 The relationship of LISP+ALT to traffic engineering seems to be
> fairly tenuous; this section seems to primarily recapitulate
> elements of the LISP architecture (not ALT) that are related to
> traffic engineering.  I would suggest removing the section entirely,
> or if not then reducing it just to the first two sentences plus a
> statement that ALT as such does not relate to traffic engineering,
> perhaps with a reference to the relevant section of [LISP].

In re-reading this section, I agree that it did a poor job of combining
two concepts (ALT stability and traffic engineering) that are only
indirectly related. I have split the discussion into two subsections
and have rewritten/expanded both to better explain the intention.

> This section glosses over the fact that although some traffic
> engineering capabilities are gained, others are lost, and that costs
> are transferred in some cases.  Depending on the scenario, some
> players in the traffic engineering equation win, others lose.
> Notably, transit providers can be seen as losing.  For transit
> traffic, if all traffic flows toward well-aggregated prefixes, the
> ability of transit SPs to engineer those flows by making
> fine-grained adjustments to routing is hampered.  Elsewhere in the
> LISP document set it is suggested that this can be addressed by
> having the transit provider re-encapsulate the transit traffic.
> While this may enable the transit provider to regain egress
> selection TE control, it comes at the cost of deploying new
> encapsulating and decapsulating routers at the transit provider
> border.  Further, current routing-based TE can provide a modicum of
> ingress selection control; no replacement is offered for this.
> There are further changes in the "balance of power" between traf fic
> sinks, transits, and traffic sources as well.  Thus, while I would
> agree that the TE capabilities provided by LISP are better in some
> ways, they may be seen as worse in others, and the "improved traffic
> engineering" claims of this section are incomplete without a
> discussion of the tradeoffs.

This is an area of the LISP effort that is in the midst of a great deal
of evolution right now. The authors are actively working to engage the
operational community in a dicussion to figure out what traffic engineering
capabilities are needed and how best to incorporate those capabilities into
LISP and the LISP mapping database. If you're interested in participating
in these discussions, I'd very much welcome your input.

> S8.2 and elsewhere mention that "It is again worth noting that the
> ALT carries only EID-prefixes, used to construct BGP paths to their
> owning ETRs; this set of information is considerably less volatile
> than the actual EID-to-RLOC mappings."  It's not clear to me why it
> should be much less volatile.  Presumably the mapping changes
> whenever an RLOC is (permanently) added or deleted.  Equally, one
> would assume that since each RLOC corresponds to an ETR, when an
> RLOC is added the ETR must be advertised into the ALT, and when an
> RLOC is deleted the ETR advertisement must be removed.  Thus the
> naive analysis is that the ALT routes should be at least equally
> volatile (but actually more so because the routing will reflect the
> dynamic state of the ETR connectivity).  Perhaps Map Servers are
> supposed to help with this?  Or maybe the point is that aggregation
> should hide the volatility close to the source?  In any case, the
> statement as written doesn't seem to be quite right.

Rewritten to clarify how and why the ALT can help improve stability.

> The various discussions of aggregation seem to be lacking in detail
> considering that aggregation is said to be key to this scheme.  In
> particular, there is no mention of whether and when to suppress
> more-specifics.  I suppose it may be assumed that you always do so?
> This needs to be made clear.  Diagrams would help.

I believe more details will come out of development of the ALT deployment
document.

> S11.3 the citation for S-BGP (not sBGP) is wrong.

Fixed.

> S13, References.  Surely [LISP] must be a normative reference?
> [LISP-MS] also seems to be.

Fixed. The distinction between "normative reference" and "informative
reference" in IETF procedural-speak is somewhat lost on me.

> The final paragraph of S.2 summarizes most of the following sections
> other than S.7 and sections 10+.  Seemed odd to omit those.

Fixed. Probably an oversight as additional sections were added.

> The definition of RLOC should probably mention that "locator" is
> used interchangeably with "RLOC".  Or, the uses of "locator" could
> be replaced by "RLOC".

Fixed.

> S.4, the first use of the acronym "DA" is not defined.  (Seems to
> have come from some c-n-p of paragraphs in the latest revision,
> since the term is defined later.)

Eliminated the acronym use and spelled-out "destination IP address".

> In section 5, "A LISP+ALT router near the edge learns EID prefixes
> originated by authoritative ETRs".  You haven't defined "the edge"
> but shouldn't this be *at* the edge?  (If you really mean "near"
> then what is "the edge", and how near is near?)

Yes, it is "at the edge". Fixed, though I haven't added a specific
definition of "the edge".

> S5.1 talks about Data Probes without mentioning that they're discouraged.

Fixed.

> S5.2, first paragraph, "The ALT network is built in a
> tree-structured hierarchy".  S8 later says "The ALT BGP peering
> topology should be arranged in a tree-like fashion (with some
> meshiness)".  The latter makes more sense than the former for
> practical purposes, but anyway they need to be reconciled.

Fixed. Good catch, thanks.

> S6.1 -- I think you could s/explicitly//g and increase clarity.  If
> you want to be emphatic, IMO you'd be better off using 2119
> terminology.

Mostly removed except where I thought the emphasis was needed.

> S9.1 "TCP-connected ETRs".  These are not defined.

Obsolete reference, removed.

> The [Interworking] reference has expired.

Updated reference.

> In addition, the following minor changes.  This is against the
> version that was sent to [email-masked] on Feb 23:
> 
> jgs$ diff draft-ietf-lisp-alt-03.txt draft-ietf-lisp-alt-03-marked-up.txt
> 250c250
> <    It is important to note that the ALT does not distibute actual EID-
> ---
> >   It is important to note that the ALT does not distribute actual EID-
> 352c352
...

Is there some context missing from this list of diffs? I can't see any
differences in the text other than spacing.

	--Vince

---

From: Vince Fuller <[email-masked]>
Date: Mon, 8 Mar 2010 13:13:57 -0800

> LISP control messages, there are additional defined fields.

FYI, as Darrel pointed out in private email, I typod the port numbers
above. They should be 4341 (LISP user data) and 4342 (LISP control).

	--Vince

---

From: "John G. Scudder" <[email-masked]>
Date: Sun, 21 Mar 2010 18:36:16 -0400

Vince,

Thanks for your reply.  I'll look over the revised doc as soon as I can; in the meantime a couple of quick comments.

> for further study as EID/RLOC separation technology evolves.

I'd be satisfied if the document pointed out that users will need to acquire their EID space from somewhere and will be tied to that supplier unless they renumber their site.  I do agree that the specifics of the business (or non-profit, or whatever) relationship between the site and the EID supplier is out of scope.  What I think is important is that a fresh reader not mistakenly think that LISP eliminates all such relationships making sites completely fancy-free.  I think this is a nuance that could be fairly easily overlooked by someone new to the topic.

If you think the tie to the EID supplier is not fundamental, I'd be very interested in hearing why.

> reference" in IETF procedural-speak is somewhat lost on me.

I'm hardly an authority but my heuristic is that if I can understand and implement the spec without looking at the reference, the reference is informative.  If I need to look at the referenced document, it's normative.

Regards,

--John

---

From: Vince Fuller <[email-masked]>
Date: Mon, 22 Mar 2010 13:45:30 -0700

Hi John-

> I'd be satisfied if the document pointed out that users will need to
> acquire their EID space from somewhere and will be tied to that
> supplier unless they renumber their site.  I do agree that the
> specifics of the business (or non-profit, or whatever) relationship
> between the site and the EID supplier is out of scope.  What I think
> is important is that a fresh reader not mistakenly think that LISP
> eliminates all such relationships making sites completely
> fancy-free.  I think this is a nuance that could be fairly easily
> overlooked by someone new to the topic.
> 

It will really depend on how the well-aggregated we want or expect the ALT
to be, how allocations are managed, and what relationships exist between
EID suppliers and operators of the ALT/MS infrastructure.

One could imagine, for example, a RIR/LIR-like model where an end user
could obtain EID space from either a direct provider of MS services (the
LIR analog) or from a higher-tier regional authority (the RIR analog) but
contracts for mapping service through an MS provider that is also within
that region. If the space were provided by the "LIR", then it could be
aggregated at that level. If it were provided by the "RIR" with mapping
service via the "LIR", then it might be aggregated at either of those levels.
Should the end site change MS providers, an "RIR"-based assignment could
still be aggregated by the "RIR".

I will add text pointing out that there are open issues regarding how
 aggressive aggregation can be, how the ALT/MS infrastructure will be
build and operated, etc.

> I'm hardly an authority but my heuristic is that if I can understand
> and implement the spec without looking at the reference, the
> reference is informative.  If I need to look at the referenced
> document, it's normative.

Ah, thanks, that is a useful consideration.

	--Vince