Subject: Re: [mpls] New Version Notification for draft-villamizar-mpls-forwarding-00.txt

From: "Carlos Pignataro (cpignata)" <[email-masked]>
Date: Sun, 16 Dec 2012 16:16:38 +0000

Hi, Kireeti,

This is a very nicely written document that covers important aspects.

One early comment is that, in a way, this seems to actually be multi-part document, including requirements, as well as compliance and performance testing. One early question is whether it makes sense to take each part separately (even in different WGs, with one as a BCP in mpls in other in bmwg continuation of rfc5695).

I also agree with Andy's comments about some PW specific pieces to this.

Some more specific comments, hopefully these are clear and useful:

2.1.  Forwarding Basics

Should this section also include forwarding specifics about reserved labels, and in particular RA.

2.3.  MPLS Multipath Techniques

...

   The most common multipath techniques are ECMP applied at the IP
   forwarding level, Ethernet LAG with inspection of the IP payload, and
   multipath on links carrying both IP and MPLS, where the IP header is
   inspected below the MPLS label stack.  In most core networks, the
   vast majority of traffic is MPLS encapsulated.

   In order to support an adequately even load distribution across
   multiple links, IP addresses must be used.  Common practice today is
   to reinspect the IP addresses at each LSR and use the label stack and
   IP addresses in a hash performed at each LSR.

This description might appear oversimplified, especially in the presence of BCP128. It was interesting that RFC 4928 is not cited, when it contains a whole section on current ECMP practices at http://tools.ietf.org/html/bcp128#section-2. Further, this section describes how in addition to IP addresses, some cases use the label stack (alone or in combination) as input into the hash for multipath.

2.3.1.  Pseudowire Control Word

   Within the core of a network some form of multipath is almost certain
   to be used.  Multipath techniques deployed today are likely to be
   looking beneath the label stack for an opportunity to hash on IP
   addresses.

   A pseudowire encapsulated at a network edge must have a means to
   prevent reordering within the core if the pseudowire will be crossing
   a network core, or any part of a network topology where multipath is
   used.

   Not supporting the ability to encapsulate a pseudowire with a control
   word may lock a product out from consideration.  A pseudowire
   capability without control word support might be sufficient for
   applications which are strictly both intra-metro and low bandwidth.
   However a provider with other applications will very likely not
   tolerate having equipment which can only support a subset of their
   pseudowire needs.

While I agree with this text, I wonder if it is attempting to set up a new requirement. For example, although not 2119-capitalized, "A pseudowire encapsulated at a network edge must have a means to prevent reordering within the core". PWE3 has been discussing this topic, but today's specs include PW Types for which a CW is optional. Similarly, recommendations are already at this http://tools.ietf.org/html/bcp128#section-3.

2.3.2.  Pseudowire Flow Label

...
   Any pseudowire which does not carry a flow label is in effect a
   single microflow (in [RFC2475] terms).

This is a bit of a corner-case nit, but for completeness: this is not the case for IP L2 Transport PW without its optional CW (without flow label) http://tools.ietf.org/html/rfc4447#section-4.1.

2.3.2.  Pseudowire Flow Label

2.3.3.  MPLS Entropy Label

   (see Section 2.3)

This is an editorial nit, but for readability -- it confused me to see a pointer to S2.3 within S2.3.x (as if there is an (apparent) reading loop).

3.  Questions for Suppliers

       Q#10  Are reserved labels included in the label stack hash?  They
             MUST NOT be included.

Could a citation be included for this MUST NOT? Or is this document adding this requirement? I think there was a mention in RFC 6790, but RFC4928/BCP128 says:

   Any reserved label, no
   matter where it is located in the stack, may be included in the
   computation for load balancing.  Modification of the label stack
   between packets of a single flow could result in re-ordering that
   flow.  That is, were an explicit null or a router-alert label to be

   added to a packet, that packet could take a different path through
   the network.

4.  Forwarding Compliance and Performance Testing

I wonder if this complete section should be progressed independently in a separate document.

Again, this is an useful well-written document, thanks!

-- Carlos.

On Oct 8, 2012, at 8:30 PM, Kireeti Kompella <[email-masked]<mailto:[email-masked]>> wrote:

Hi Folks,

I'd spoken on the issue of deep label stacks at the last IETF, and there was interest from chip suppliers, vendors and service providers.  Curtis just submitted the draft; it goes beyond just deep label stacks.  There are suggestions in the draft on aspects of implementing MPLS forwarding, as well as questions to ask regarding an implementation, and things to test.

Please read and comment to the list.

If you (that includes MPLS WG chairs and ADs) could also  formulate your thoughts on what type of doc (Info, BCP, PS) this should be, that would be very helpful in moving this discussion forward.

Thanks,
Kireeti.

Begin forwarded message:

From: [email-masked]<mailto:[email-masked]>
Subject: New Version Notification for draft-villamizar-mpls-forwarding-00.txt
Date: October 8, 2012 16:49:08PDT
To: [email-masked]<mailto:[email-masked]>
Cc: [email-masked]<mailto:[email-masked]>

A new version of I-D, draft-villamizar-mpls-forwarding-00.txt
has been successfully submitted by Curtis Villamizar and posted to the
IETF repository.

Filename: draft-villamizar-mpls-forwarding
Revision: 00
Title: MPLS Forwarding Compliance and Performance Requirements
Creation date: 2012-10-08
WG ID: Individual Submission
Number of pages: 17
URL:             http://www.ietf.org/internet-drafts/draft-villamizar-mpls-forwarding-00.txt
Status:          http://datatracker.ietf.org/doc/draft-villamizar-mpls-forwarding
Htmlized:        http://tools.ietf.org/html/draft-villamizar-mpls-forwarding-00

Abstract:
  This document provides guidelines for implementors regarding MPLS
  forwarding and a basis for evaluations of forwarding implementations.
  Guidelines cover basic MPLS forwarding, forwarding when a deep MPLS
  label stack is encountered, MPLS UHP operations which require one or
  more label POP plus a PUSH, guidelines for hashing an MPLS stack and
  payload for multipath, and conformance and performance requirements
  for recent pseudowire and MPLS standards.

The IETF Secretariat

_______________________________________________
mpls mailing list
[email-masked]<mailto:[email-masked]>
https://www.ietf.org/mailman/listinfo/mpls

---

From: "Andrew G. Malis" <[email-masked]>
Date: Mon, 17 Dec 2012 08:18:09 -0500

Carlos,

On Sun, Dec 16, 2012 at 11:16 AM, Carlos Pignataro (cpignata) <
[email-masked]> wrote:

>  While I agree with this text, I wonder if it is attempting to set up a
> new requirement. For example, although not 2119-capitalized, "A pseudowire
> encapsulated at a network edge must have a means to prevent reordering
> within the core". PWE3 has been discussing this topic, but today's specs
> include PW Types for which a CW is optional. Similarly, recommendations are
> already at this http://tools.ietf.org/html/bcp128#section-3
>
> .
>

I don't think this is a new requirement at all, it's consistent with the
"avoiding ECMP" section (section 2) of RFC 4385. Perhaps just a reference
to 4385 should be added here.

Cheers,
Andy

---

From: "Carlos Pignataro (cpignata)" <[email-masked]>
Date: Mon, 17 Dec 2012 16:56:40 +0000

Hi, Andy,

On Dec 17, 2012, at 8:18 AM, Andrew G. Malis <[email-masked]<mailto:[email-masked]>> wrote:

Carlos,

On Sun, Dec 16, 2012 at 11:16 AM, Carlos Pignataro (cpignata) <[email-masked]<mailto:[email-masked]>> wrote:

While I agree with this text, I wonder if it is attempting to set up a new requirement. For example, although not 2119-capitalized, "A pseudowire encapsulated at a network edge must have a means to prevent reordering within the core". PWE3 has been discussing this topic, but today's specs include PW Types for which a CW is optional. Similarly, recommendations are already at this http://tools.ietf.org/html/bcp128#section-3

.

I don't think this is a new requirement at all, it's consistent with the "avoiding ECMP" section (section 2) of RFC 4385. Perhaps just a reference to 4385 should be added here.

That's a good point, a pointer to S2 of RFC 4385 could certainly help.

In reading, it appeared to me that the text in this document was a bit stronger, applicable to every pseudowire ("A PW encapsulated at a network edge must have a means to prevent reordering") as opposed to the Section from 4385 ("If a PW is sensitive to packet misordering and ...").

Thanks,

-- Carlos.

Cheers,
Andy

---

From: Kireeti Kompella <[email-masked]>
Date: Mon, 17 Dec 2012 09:42:10 -0800

Hi Carlos,

Agree that this document covers a lot of different topics.  The idea was to
float it, and gauge the level of interest.  If the WG and chairs thinks
that it should be split up, that's fine.

We will include more about handling reserved labels (aka special purpose
labels) in a future version.  Good point about RA.

Andy answered the PW questions.

The MUST NOT for hashing ECMP is (as you say) in RFC 6790.

Thanks for your comments!

Kireeti

On Sun, Dec 16, 2012 at 8:16 AM, Carlos Pignataro (cpignata) <
[email-masked]> wrote:

>  One early comment is that, in a way, this seems to actually be
> multi-part document, including requirements, as well as compliance and
> performance testing. One early question is whether it makes sense to take
> each part separately (even in different WGs, with one as a BCP in mpls in
> other in bmwg continuation of rfc5695).
>
> This description might appear oversimplified, especially in the presence
> of BCP128. It was interesting that RFC 4928 is not cited, when it contains
> a whole section on current ECMP practices at
> http://tools.ietf.org/html/bcp128#section-2. Further, this section
> describes how in addition to IP addresses, some cases use the label stack
> (alone or in combination) as input into the hash for multipath.
>
> already at this http://tools.ietf.org/html/bcp128#section-3.
>
> This is a bit of a corner-case nit, but for completeness: this is not the
> case for IP L2 Transport PW without its optional CW (without flow label)
> http://tools.ietf.org/html/rfc4447#section-4.1.
>
> This is an editorial nit, but for readability -- it confused me to see a
> pointer to S2.3 within S2.3.x (as if there is an (apparent) reading loop).
>
> Could a citation be included for this MUST NOT? Or is this document adding
> this requirement? I think there was a mention in RFC 6790, but
> RFC4928/BCP128 says:
>
>     Any reserved label, no
>  On Oct 8, 2012, at 8:30 PM, Kireeti Kompella <[email-masked]>
> wrote:
>
>  Hi Folks,
>
>  I'd spoken on the issue of deep label stacks at the last IETF, and there
> was interest from chip suppliers, vendors and service providers.  Curtis
> just submitted the draft; it goes beyond just deep label stacks.  There are
> suggestions in the draft on aspects of implementing MPLS forwarding, as
> well as questions to ask regarding an implementation, and things to test.
>
>  If you (that includes MPLS WG chairs and ADs) could also  formulate your
> thoughts on what type of doc (Info, BCP, PS) this should be, that would be
> very helpful in moving this discussion forward.
>
>  Thanks,
> Kireeti.
>
> Begin forwarded message:
>
>  *From: [email-masked]
>  *Subject: **New Version Notification for
> draft-villamizar-mpls-forwarding-00.txt*
>  *Date: *October 8, 2012 16:49:08PDT
>  *To: [email-masked]
>  *Cc: [email-masked]
>
>
> URL:
> http://www.ietf.org/internet-drafts/draft-villamizar-mpls-forwarding-00.txt
> Status:
> http://datatracker.ietf.org/doc/draft-villamizar-mpls-forwarding
> Htmlized:
> http://tools.ietf.org/html/draft-villamizar-mpls-forwarding-00
>
>
> Abstract:
> [email-masked]

-- 
Kireeti

---

From: Curtis Villamizar <[email-masked]>
Date: Wed, 19 Dec 2012 19:08:02 -0500

Hi Carlos,

I see Andy and Kireeti already replied.

In message <[email-masked]>
"Carlos Pignataro (cpignata)" writes:
> performance testing. One early question is whether it makes sense to
> take each part separately (even in different WGs, with one as a BCP in
> mpls in other in bmwg continuation of rfc5695).
>  

The sections with questions to ask and testing are intentionally terse
and serve only as outlines.  This is to put a summary in one place.
If BMWG wants to launch one or more documents to cover the testing in
detailed benchmark specifications, they are more than welcome to do
so.  That is why we gave Al a heads up and why Al gave the BMWG
mailing list a heads up.

Of course, if there is consensus to break up this document into more
than one, I'm OK with that approach.

> labels, and in particular RA.

... what Kireeti said.

Yes we could reference the reserved label list and also if there is
new work on "special purpose labels" we can reference that work too.

So far RFC 3032 specifies RA and the NULL labels (RFC 3032 is
obviously cited), the update to NULL is in RFC 4182 (cited), and GAL
is defined in RFC 5586.  We also cover EL and ELI (RFC 6790) in some
detail.  At this point that covers it for reserved labels.  We can
explicitly point out RA.

We didn't cover OAM Alert Label #14 RFC3429 as use of ITU-T Y.1711 is
strongly discouraged today.  Y.1711 doesn't work at all for multipath
and all service providers use multipath.

When Kireeti's "special purpose labels" draft becomes a WG item we can
point to "special purpose labels".  I'm not sure what the WG concensus
is on that.  We still have 4-6 and 8-12 to assign, which should be
plenty IMHO, but the extension mechanism can't hurt (too much).

Its been pointed out in private email that some TP OAM functions
require hardware support (DM, LM, and to a lesser extent CC/CV).  The
most challenging may be LM.  Also PTP and NTP benefit from hardware
support and the TICTOC work should be reflected here.

> This description might appear oversimplified, especially in the
> presence of BCP128. It was interesting that RFC 4928 is not cited,
> when it contains a whole section on current ECMP practices at
> describes how in addition to IP addresses, some cases use the label
> stack (alone or in combination) as input into the hash for multipath.

Both RFC4928 (BCP128) and RFC4385 are cited.  RFC4928 section 2 is a
little dated and represents the LCD in roughly 2004 or so when this
draft was first introduced.

In response to private email comments from Shane Anante the
paragraph you pointed out as oversimplified had already been expanded.
A lot more is now said about the so-called 5-tuple commonly used and
the use of other fields.  There is now a new section "2.3.5.  Fields
Used for Multipath" which we can put a forward reference to.

> new requirement. For example, although not 2119-capitalized, "A
> pseudowire encapsulated at a network edge must have a means to prevent
> reordering within the core". PWE3 has been discussing this topic, but
> today's specs include PW Types for which a CW is optional. Similarly,
> recommendations are already at this
> http://tools.ietf.org/html/bcp128#section-3.

What Andy said ...

BTW- Andy has sent quite a lot of private email about PW which will be
reflected in the next iteration.

> This is a bit of a corner-case nit, but for completeness: this is not
> the case for IP L2 Transport PW without its optional CW (without flow
> label) http://tools.ietf.org/html/rfc4447#section-4.1.

People use L2TP?  :-)

OK - PW with MPLS PSN, but then again this is an MPLS document.

I don't see what point you are trying to make.

> This is an editorial nit, but for readability -- it confused me to see
> a pointer to S2.3 within S2.3.x (as if there is an (apparent) reading
> loop).

I'll change the reference to "2.3.5.  Fields in Multipath" which
is new.  BTW- that section starts by saying "Inspecting the IP payload
provides the most entropy in provider networks.  The practice of
looking past the bottom of stack label for an IP payload is well
accepted and documented in [RFC4928] and in other RFCs." just to be
clear that the reason for inclusion in an MPLS document is that you
need to look under the MPLS label stack for this purpose.

> Could a citation be included for this MUST NOT? Or is this document
> adding this requirement? I think there was a mention in RFC 6790, but
> RFC4928/BCP128 says:

Eliminated the double negative.  It now says "Are reserved labels
excluded from the label stack hash?  They MUST be excluded [RFC6790]."

RFC 6790 is cited elsewhere but is now cited here as well.

> I wonder if this complete section should be progressed independently

If your "wondering" becomes the WG concensus, then I'm OK with that.

My preference is to keep this brief summary of necessary testing with
the requirements and expand on details of specific tests in BMWG.

> Again, this is an useful well-written document, thanks!

Thanks for the comments, here in MPLS as well as in BMWG.

The following changes have been edited into (my local copy) of the
draft and are likely to make the next iteration (unless Kireeti
objects).

  1.  Expanded a bit on reserved labels.  Mention TP OAM (to be
      expanded on in later sections).  Mention PTP and NTP.

  2.  Defer inclusion of "special purpose labels" for now.

  3.  The paragraph introducing multipath had already been expanded.

  4.  Andy already sent two rounds of extensive comments on PW.  I
      have yet to edit in the second round.

  5.  Clarified PW without flow label to mean PW carried over MPLS
      without flow label (though I don't really agree that we need
      that clarification).

  6.  Fixed reference to 2.3 in 2.3.3.  It now refers to the new
      section "2.3.5.  Fields in Multipath".

  7.  Changed "MUST NOT be included" to "MUST exclude" in two places
      where reserved labels in hash are discussed.  Referenced RFC6790
      in both places.

I may send out a 01 without Andy's latest rounds of comments, then
following shortly after with an 02.  This would be a better basis for
discussion.  I'll see what Kireeti thinks of sending this out vs
waiting for all of the changes to get edited in and then sending out
an 01.  We still have off list discussions on other topics such as TP
OAM with changes pending.

Curtis

> I'd spoken on the issue of deep label stacks at the last IETF, and
> there was interest from chip suppliers, vendors and service providers.
> Curtis just submitted the draft; it goes beyond just deep label
> stacks.  There are suggestions in the draft on aspects of implementing
> MPLS forwarding, as well as questions to ask regarding an
> implementation, and things to test.
>  
> If you (that includes MPLS WG chairs and ADs) could also formulate
> your thoughts on what type of doc (Info, BCP, PS) this should be, that
> would be very helpful in moving this discussion forward.
>  
> Thanks,
> Kireeti.

---

From: "Carlos Pignataro (cpignata)" <[email-masked]>
Date: Thu, 20 Dec 2012 15:49:01 +0000

Hi, Curtis,

Thanks for the response -- please see inline, skipping responses as implicit Ack.

> so.  That is why we gave Al a heads up and why Al gave the BMWG

One concern with the terse specification of testing cases is that they could use quite different methodology -- for example, how is the determination required in T#10 performed?

> reflected in the next iteration.

Thanks.

>> label) http://tools.ietf.org/html/rfc4447#section-4.1.

Very much so -- but that is orthogonal to my comment, which is about IP L2 and not L2TP.

> I don't see what point you are trying to make.

Sorry if I managed to confuse you. Let me try again -- it is a nit.

RFC 4447 (PW setup using LDP) defines in Section 4.1 (pointer included above) the "IP Layer 2 Transport" Pseudowire (PW Type 0x000B). This is a PW FEC, that when used without CW looks in the DP as an RFC 3032 MPLS encapsulated IP datagram. It's used in arp-med, ipls, etc.

I was just pointing out that for that particular PW, the multipath is as if IP payload directly on MPLS (since it is in the DP). I do not intend to convolute the description and remove generalization to somehow cover this, but I thought I'd bring it up in case.

> need to look under the MPLS label stack for this purpose.

Thanks.

> OAM with changes pending.

Thanks much.

-- Carlos.

---

From: Curtis Villamizar <[email-masked]>
Date: Thu, 20 Dec 2012 14:21:58 -0500

Carlos,

See inline.

In message <[email-masked]>
"Carlos Pignataro (cpignata)" writes:
> One concern with the terse specification of testing cases is that they
> could use quite different methodology -- for example, how is the
> determination required in T#10 performed?

This document is not about testing methodology.  Testing methodology
should be specified in BMWG (if at all).

This document needs only to say "You [chip vendor, system vendor,
provider] need to 'text for X'".  Only in cases were some might question
whether such testing is even feasible should there be any mention of
how to "test for X", and maybe not even then.

Today most (or all) service providers do extensive unpublished testing
of product (plus field trial) before deployment.  The real world has a
habit of producing far more challenging test cases than BMWG has and
that is why equipment which benchmarks well often fails or performs
poorly in service provider testing, and sometimes in field testing or
worse yet in the field.

In the absense of BMWG testing methodologies it is up to the tester
(test organization as a whole) to figure out a way to "test for X".
Even where BMWG testing methodologies exist it is up to the tester to
figure out whether BMWG testing methodologies adequately "test for X".

> Very much so -- but that is orthogonal to my comment, which is about

The point I was making was that this is an MPLS document.

> RFC 4447 (PW setup using LDP) defines in Section 4.1 (pointer included
> above) the "IP Layer 2 Transport" Pseudowire (PW Type 0x000B). This is
> a PW FEC, that when used without CW looks in the DP as an RFC 3032
> MPLS encapsulated IP datagram. It's used in arp-med, ipls, etc.
>  
> I was just pointing out that for that particular PW, the multipath is
> as if IP payload directly on MPLS (since it is in the DP). I do not
> intend to convolute the description and remove generalization to
> somehow cover this, but I thought I'd bring it up in case.

AFAIK no one uses the LDP FEC to determine whether the payload is IP
or not.  It is also entirely useless if LDP is carried in RSVP-TE LSP
as is often the case.

This part of RFC 4447 says "The PW control word MAY be inserted
between the MPLS label stack and the IP payload."

If load balancing is desired, then either 1) don't insert a CW, or 2)
add an entropy label, or 3) both.

If If load balancing is *not* desired, then either 1) add a CW and
either don't add an entropy label, or 2) add an entropy label with a
fixed value for all packets in the PW. or 3) both.

If you do this (above two paragraphs) then load split works exactly as
intended in RFC4385 and RFC4928.

Maybe this needs to be mentioned somewhere.

> > OAM with changes pending.

Thanks for clarifying your prior comment on IP PW.  I'll look for a
place in the document to put the text on IP PW if this addresses your
comment.

Curtis

---

From: "Carlos Pignataro (cpignata)" <[email-masked]>
Date: Thu, 20 Dec 2012 20:49:10 +0000

It does -- thanks for addressing.

-- Carlos.

On 12/20/12 3:48 PM, "Curtis Villamizar" <[email-masked]> wrote:

>
>Carlos,
>
>Revisiting one nit of yours.  Trimmed the rest.
>
>In message 
><[email-masked]>
>OK I finally see the nit that started out this part of the thread.
>
>Here is the new text for the offending paragraph (xml source).
>
>          <t>
>            <!-- nit pointed out by Carlos -->
>            Any pseudowire carried over MPLS which makes use of the
>            pseudowire control word and does not carry a
>            flow label is in effect a single microflow (in <xref
>            target="RFC2475" /> terms).
>          </t>
>
>I think this covers the original nit regarding the corner case of IP
>PW.  If an IP PW or any other PW has a CW and no flow label it is
>effectively treated as one microflow in RFC2475 terms.  [If it is not
>an IP PW and has no control word it is going to be a mess of reordered
>PW traffic, even with flow label.]
>
>Curtis
>

---

From: Curtis Villamizar <[email-masked]>
Date: Thu, 20 Dec 2012 15:48:11 -0500

Carlos,

          <t>

Curtis