Subject: Re: [RTG-DIR] Rtgdir last call review of draft-ietf-detnet-mpls-04

From: Balázs Varga A <[email-masked]>
Date: Tue, 7 Jan 2020 14:32:56 +0000

Hi Carlos/Stewart,
Thanks for the comments/nits and starting to sort them out.
We have started to work on the received comment.
I will be back with a follow-up on these and the rest items as well.  
Many thanks & Cheers
Bala'zs

-----Original Message-----
From: Carlos Pignataro (cpignata) <[email-masked]> 
Sent: Tuesday, December 24, 2019 10:16 PM
To: Stewart Bryant <[email-masked]>
Cc: Routing Directorate <[email-masked]>; [email-masked]; [email-masked]; [email-masked]
Subject: Re: Rtgdir last call review of draft-ietf-detnet-mpls-04

Hi, Stewart,

> 2019/12/24 午前9:26、Stewart Bryant <[email-masked]>のメール:
> 
> 
> 
>> On 24 Dec 2019, at 00:03, Carlos Pignataro via Datatracker <[email-masked]> wrote:
>> 
>> 
>> 3.1.  Layers of DetNet Data Plane
>> 
>>   The DetNet control word (d-CW)
>>   conforms to the Generic PW MPLS Control Word (PWMCW) defined in
>>   [RFC4385].
>> 
>> Yes, but why not the Preferred CW?
>> 
> 
> Hi Carlos
> 
> Thanks for the review.

Anytime!

> Just picking up a couple of points at this stage,

Sure, happy to follow-up on the rest at a later time.

> 
> The PCW only supports a 16bit sequence number and it has the skip zero auto-signalling of active S/N feature.
> 
> This was a problem for DetNet because:
> 
> - We were worried about S/N rollover frequency in some applications and so we wanted the option of a larger S/N.
> 
> - We wanted to have the option to propagate the S/N from the payload to the transport to simplify the implementation in some cases. These applications have a non-skip zero S/N. Skip zero is an irritation to implement and we should probably have signalled in in PWs.
> 
> As you note in is only a preferred design for PWs, DetNet is not 
> constrained by that and there were good reasons to adopt this 
> alternate approach,
> 
> 

My main point is that, all that you say above, is not in the draft. The spec seems underspecified — including what’s in-order, out-of-order, which specific application layer (or some higher layer) sequencing schemes might map to, etc.

> 
>> 4.1.  DetNet Over MPLS Encapsulation Components
>> 
>>   The LSP used to forward the DetNet packet may be of any type (MPLS-
>>   LDP, MPLS-TE, MPLS-TP [RFC5921], or MPLS-SR
>>   [I-D.ietf-spring-segment-routing-mpls]).
>> 
>> I am not sure of the value of this statement for an "MPLS Dataplane" document.
>> Further, are these "LSP Types" and if so where are the different 
>> types ennumerated? Does this mean that static binding LSPs and BGP 
>> signaled cannto be used? T-LDP does not work? "SDN Assigned"? I 
>> recommend removing this, since it can confuse and does not add much.
> 
> I think it just needs a “for example” and I think that it needs to be made clear that the design is not restricted to a single method of establishing an LSP nor to the characteristics and constraints that go with those LSPs.

It would be a good start to made that clear and qualify the list. Still, what are “LSP Types”? Does it really add to ennumerate some and not others, even if “for-example”ing it?

> 
>> 
>> 4.3.  OAM Indication
>> 
>> It is important to have the OAM Indication, but what type of OAM 
>> packets can run on top of this AcH? I found it interesting that for 
>> example there is not reference or citation to RFC 8029.
>> 
> 
> Work has started on OAM, for example draft-mirsky-detnet-mpls-oam-00
> 
> The OAM for DetNet will be more complex than the OAM for a classical P2P or P2MP LSP (or PW) because of the PREOF function. There is nothing in the data plane that precludes us using on of the existing OAM indicators (GAL or 0001 ACH), but I think that it is important to thing through the subtleties. Thus I think it is OK to make progress on the elements of the data plane that we can nail down, and leave. The OAM as follow-up work.
> 

I believe being explicit rather than implicit about scoping things out adds to clarity.

Do you expect an equal number of DetNet documents concerning themselves with OAM, as there are dataplane docs?

> 
>> 
>> 
>> Also, outdated reference: draft-ietf-spring-segment-routing-mpls has 
>> been published as RFC 8660
> 
> This will get picked up on the next resin as part of the Nits check.
> 

Thanks!

Happy Holidays :-)

Carlos.

> Best regards
> Stewart

---

From: Balázs Varga A <[email-masked]>
Date: Tue, 7 Jan 2020 14:36:24 +0000

Hi Sasha,
Thanks for pointing to this topic.
We will discuss how to make “in-order” and “out-of-order” much more clear and update the text.
Thanks
Bala’zs

From: Alexander Vainshtein <[email-masked]>
Sent: Tuesday, December 24, 2019 3:35 PM
To: Stewart Bryant <[email-masked]>; Carlos Pignataro <[email-masked]>
Cc: [email-masked]; [email-masked]; [email-masked]; [email-masked]
Subject: RE: [RTG-DIR] Rtgdir last call review of draft-ietf-detnet-mpls-04

Stewart, Carlos and all,
I understand the concerns with the 16-bit space for sequence numbers and the skip-zero approach taken by RFC 4385 (presumably, for compatibility with the early Martini drafts that have already been implemented). And it was quite convenient to turn off sequencing in PWs simply by setting SN to zero,

(The TDM PWs that MUST use sequence numbers because they MUST detect lost packets and compensate the lost payload use the entire 16-bit SN space).

But  I think that the draft should clearly articulate these considerations and also provide a definition of in-order and out-of-order packets. I have not found any of these things in the draft – did I miss something?

Regards,
Sasha

Office: +972-39266302
Cell:      +972-549266302
Email:   [email-masked]<mailto:[email-masked]>

From: rtg-dir <[email-masked]<mailto:[email-masked]>> On Behalf Of Stewart Bryant
Sent: Tuesday, December 24, 2019 4:26 PM
To: Carlos Pignataro <[email-masked]<mailto:[email-masked]>>
Cc: [email-masked]<mailto:[email-masked]>; [email-masked]<mailto:[email-masked]>; [email-masked]<mailto:[email-masked]>; [email-masked]<mailto:[email-masked]>
Subject: Re: [RTG-DIR] Rtgdir last call review of draft-ietf-detnet-mpls-04

On 24 Dec 2019, at 00:03, Carlos Pignataro via Datatracker <[email-masked]<mailto:[email-masked]>> wrote:

Hi Carlos

Thanks for the review. Just picking up a couple of points at this stage,

As you note in is only a preferred design for PWs, DetNet is not constrained by that and there were good reasons to adopt this alternate approach,

I am not sure of the value of this statement for an "MPLS Dataplane" document.
Further, are these "LSP Types" and if so where are the different types
ennumerated? Does this mean that static binding LSPs and BGP signaled cannto be
used? T-LDP does not work? "SDN Assigned"? I recommend removing this, since it
can confuse and does not add much.

4.3.  OAM Indication

It is important to have the OAM Indication, but what type of OAM packets can
run on top of this AcH? I found it interesting that for example there is not
reference or citation to RFC 8029.

Also, outdated reference: draft-ietf-spring-segment-routing-mpls has been
published as RFC 8660

Best regards

Stewart

___________________________________________________________________________

This e-mail message is intended for the recipient only and contains information which is
CONFIDENTIAL and which may be proprietary to ECI Telecom. If you have received this
transmission in error, please inform us by e-mail, phone or fax, and then delete the original
and all copies thereof.
___________________________________________________________________________

---

From: Balázs Varga A <[email-masked]>
Date: Fri, 10 Jan 2020 13:56:22 +0000

Hi Carlos,

Many thanks for the review and your mailing with Stewart. 
Your former discussion points I tried to incorporate in this mail. 
Proposed resolutions are described below (marked with 
"<Balazs> and/or <Stewart>").

We will start to update the text if reactions are OK with You.

Thanks & Cheers
Bala'zs

-----Original Message-----
From: Carlos Pignataro via Datatracker <[email-masked]> 
Sent: Tuesday, December 24, 2019 1:03 AM
To: [email-masked]
Cc: [email-masked]; [email-masked]; [email-masked]
Subject: Rtgdir last call review of draft-ietf-detnet-mpls-04

Reviewer: Carlos Pignataro
Review result: Has Issues

Hello,

I have been selected as the Routing Directorate reviewer for this draft. The Routing Directorate seeks to review all routing or routing-related drafts as they pass through IETF last call and IESG review, and sometimes on special request. The purpose of the review is to provide assistance to the Routing ADs.
For more information about the Routing Directorate, please see ​http://trac.tools.ietf.org/area/rtg/trac/wiki/RtgDir

Although these comments are primarily for the use of the Routing ADs, it would be helpful if you could consider them along with any other IETF Last Call comments that you receive, and strive to resolve them through discussion or by updating the draft.

Document: draft-ietf-detnet-mpls-04
Reviewer: Carlos Pignataro
Review Date: December 2019
Intended Status: Proposed Standard

Summary:

I have some minor concerns about this document that I think should be resolved before publication.

[Resolving, in this context, means discussing and responding to, some of these would and others would not imply changes]

This document specifies the Deterministic Networking data plane when operating over an MPLS Packet Switched Networks.

Major Issues:

Minor Issues:

The document lists 7 (i.e, more than 5) authors.

A am somewhat confused regarding how some of these DETNET dataplane documents interact, interface, and intersect with each other. Where are requirements ultimately coming from for any variety of dataplanes? Specifically: *
draft-ietf-detnet-data-plane-framework-03 targets Informational (should be
STD?) but seems to include foundational Reqs. * draft-ietf-detnet-ip for IP * draft-ietf-detnet-ip-over-mpls for IP over MPLS * draft-ietf-detnet-ip-over-tsn and draft-ietf-detnet-mpls-over-tsn for X over TSN * draft-ietf-detnet-mpls-over-udp-ip and draft-ietf-detnet-mpls for MPLS Among all of these, where are really the requirements for MPLS? and for IP? Is there a "Roadmap" or Rosetta Stone to understand these? There are many permutations of X-over-Y. I appreciate that, not having followed DETNET discussions, I am likely missing something.

<Balazs> One can group the drafts in three sets:
1, General overview
The "draft-ietf-detnet-data-plane-framework" is intended to be used as a "Rosetta Stone".
It provides an overall framework for the Deterministic Networking data plane. Framework covers 
concepts and considerations that are generally common to any Deterministic Networking data 
plane specification.
2, DetNet Data Plane specifications
Two drafts are dedicated to IP and to MPLS based DetNet data planes. These drafts specifies
the requirements for MPLS and for IP.  "draft-ietf-detnet-mpls" specifies the Deterministic 
Networking data plane when operating over an MPLS Packet Switched Networks. 
"draft-ietf-detnet-ip" specifies the Deterministic Networking data plane when operating 
in an IP Packet Switched Networks.
3, Specifications for X-over-Y scenarios
Remaining drafts cover the X-over-Y scenarios. Some are called as "sub-network" scenarios
for DetNet. For example, if DetNet MPLS nodes are interconnected by and IP sub-network,
what is described "draft-ietf-detnet-mpls-over-udp-ip". 

<Balazs>/<Stewart> Sum of mailing + proposed fixing:
The PCW only supports a 16bit sequence number and it has the skip zero auto-signaling of active S/N feature.
This was a problem for DetNet because:
- We were worried about S/N rollover frequency in some applications and so we wanted the option of a larger S/N.
- We wanted to have the option to propagate the S/N from the payload to the transport to simplify the implementation 
in some cases. These applications have a non-skip zero S/N. Skip zero is an irritation to implement and we should probably 
have signaled in in PWs.
As you note in is only a preferred design for PWs, DetNet is not constrained by that and there were good reasons to adopt 
this alternate approach.
We assume to fix this with adding above information to the text.
NEW text to be added in section 4.2.1:
    "This format of the d-CW was created in order (1) to allow larger S/N space to 
    avoid S/N rollover frequency in some applications and (2) to allow non-skip 
    zero S/N what simplifies implementation."

I am not sure of the value of this statement for an "MPLS Dataplane" document.
Further, are these "LSP Types" and if so where are the different types ennumerated? Does this mean that static binding LSPs and BGP signaled cannto be used? T-LDP does not work? "SDN Assigned"? I recommend removing this, since it can confuse and does not add much.

<Balazs>/<Stewart> Text intends to say that any type of LSP can be used below DetNet. 
DetNet design is not restricted to a single method of establishing an LSP nor to the 
characteristics and constraints that go with those LSPs.
We assume to fix this with adding “for example” to the text and avoid the term "type".
CHANGE text to:
   "The LSP used to forward the DetNet packet is not restricted regarding any method 
   used for establishing that LSP (for example, MPLS-LDP, MPLS-TE, MPLS-TP [RFC5921],
   MPLS-SR [RFC8660], etc.)."

4.3.  OAM Indication

It is important to have the OAM Indication, but what type of OAM packets can run on top of this AcH? I found it interesting that for example there is not reference or citation to RFC 8029.

<Balazs>/<Stewart> Sum of former mailing + proposed fixing
Work has started on OAM for DetNet. We have two drafts dealing with IP and with MPLS based DetNet data planes (draft-mirsky-detnet-ip-oam-01, 
draft-mirsky-detnet-mpls-oam-01). We expect to cover OAM for DetNet with these two OAM documents.
The OAM for DetNet will be more complex than the OAM for a classical P2P or P2MP LSP (or PW) because of the PREOF function. There is nothing in the data 
plane that precludes us using on of the existing OAM indicators (GAL or 0001 ACH), but I think that it is important to thing through the subtleties. Thus I think 
it is OK to make progress on the elements of the data plane that we can nail down, and leave. The OAM as follow-up work.
We assume to fix issue this with adding reference to DetNet OAM drafts.
Add text to section 4.3:
   "DetNet specific OAM functions for MPLS based DetNet data plane are discussed 
     in [draft-mirsky-detnet-mpls-oam-01]."

Nits:

2.2.  Abbreviations

   The following abbreviations are used in this document:

Many of these abbreviations are well-known, many others have authoritative definitions and expansions. I believe this section should point to the appropriate RFCs for LSR, CW, PE, OAM, etc...
Also, interesting, this section does not expand LSE, and I think it should use it.

<Balazs> OK. We will update.

4.6.1.  Class of Service

Should this include "and TTL" in the title? I was looking for TTL specs in the Table of Contents but could not find it.

<Balazs> There are no new TTL related procedures defined for DetNet. 
Regarding TTL the section just intends to say that existing stuff can be used.
So, I think we can leave the title as it is (no reference to TTL)

9.2.  Informative References

Are all the "draft-ietf-detnet-*" really Informative?
<Balazs> No, we will fix and update.

Also, outdated reference: draft-ietf-spring-segment-routing-mpls has been published as RFC 8660
<Balazs>/<Stewart> OK, We will update.

   [RFC8655]  Finn, N., Thubert, P., Varga, B., and J. Farkas,
              "Deterministic Networking Architecture", RFC 8655,
              DOI 10.17487/RFC8655, October 2019,
              <https://protect2.fireeye.com/v1/url?k=7a40605f-26946939-7a4020c4-8610d8a762ca-97dd584868114772&q=1&e=42fa17eb-44f5-42d1-9898-4d6ba7365c47&u=https%3A%2F%2Fwww.rfc-editor.org%2Finfo%2Frfc8655>.

And is this also Informational?
<Balazs> No, We will update.

Thank you!

Carlos Pignataro.

---

From: "Carlos Pignataro (cpignata)" <[email-masked]>
Date: Fri, 10 Jan 2020 15:11:47 +0000

Thanks for the response, Balázs. Please see inline.

> 2020/01/10 午前8:56、Balázs Varga A <[email-masked]>のメール:
> 
> Hi Carlos,
> 

Maybe a question not only for you — but: if this is a fundamental document, how is it Informational?

> what is described "draft-ietf-detnet-mpls-over-udp-ip". 

I have not looked at the details, but this sounds a bit strange. You might end up with an expanding permutation of drafts and gaps. Why not include the X-over-Y in the Y document?

>    zero S/N what simplifies implementation."

Sounds good.

>   MPLS-SR [RFC8660], etc.)."

OK.

>     in [draft-mirsky-detnet-mpls-oam-01]."

I disagree. This is an individual draft. I’d add “For further study” without pointing to any early individual documents.

Thanks!

Carlos.

---

From: Balázs Varga A <[email-masked]>
Date: Mon, 13 Jan 2020 14:51:42 +0000

Hi Carlos,

Many thanks for the feedback. Regarding the 2 remaining open topics:
- Referencing the framework document: We intend to keep it "informational" as these references are in the text “background” and “e.g.,”.
- OAM: We will change text to highlight that OAM is for further study. No reference to an individual draft.  " ... additional considerations on OAM are further study ..."

I will start to update the draft accordingly.

Thanks & Cheers
Bala'zs

-----Original Message-----
From: Carlos Pignataro (cpignata) <[email-masked]> 
Sent: Friday, January 10, 2020 4:12 PM
To: Balázs Varga A <[email-masked]>
Cc: Routing Directorate <[email-masked]>; [email-masked]; [email-masked]; [email-masked]
Subject: Re: Rtgdir last call review of draft-ietf-detnet-mpls-04

> Proposed resolutions are described below (marked with "<Balazs> and/or 
> <Stewart>").
> 
> For more information about the Routing Directorate, please see 
> ​http://trac.tools.ietf.org/area/rtg/trac/wiki/RtgDir
> 
> [Resolving, in this context, means discussing and responding to, some 
> of these would and others would not imply changes]
> 
> A am somewhat confused regarding how some of these DETNET dataplane 
> documents interact, interface, and intersect with each other. Where 
> are requirements ultimately coming from for any variety of dataplanes? 
> Specifically: *
> It provides an overall framework for the Deterministic Networking data 
> plane. Framework covers concepts and considerations that are generally 
> common to any Deterministic Networking data plane specification.

> Two drafts are dedicated to IP and to MPLS based DetNet data planes. 
> These drafts specifies the requirements for MPLS and for IP.  
> "draft-ietf-detnet-mpls" specifies the Deterministic Networking data plane when operating over an MPLS Packet Switched Networks.
> "draft-ietf-detnet-ip" specifies the Deterministic Networking data 
> plane when operating in an IP Packet Switched Networks.
> 3, Specifications for X-over-Y scenarios Remaining drafts cover the 
> X-over-Y scenarios. Some are called as "sub-network" scenarios for 
> DetNet. For example, if DetNet MPLS nodes are interconnected by and IP 
> sub-network, what is described "draft-ietf-detnet-mpls-over-udp-ip".

> - We wanted to have the option to propagate the S/N from the payload 
> to the transport to simplify the implementation in some cases. These 
> applications have a non-skip zero S/N. Skip zero is an irritation to implement and we should probably have signaled in in PWs.
> constrained by that and there were good reasons to adopt this alternate approach.

Sounds good.

> DetNet design is not restricted to a single method of establishing an 
> LSP nor to the characteristics and constraints that go with those LSPs.

OK.

> <Balazs>/<Stewart> Sum of former mailing + proposed fixing Work has 
> started on OAM for DetNet. We have two drafts dealing with IP and with 
> MPLS based DetNet data planes (draft-mirsky-detnet-ip-oam-01, draft-mirsky-detnet-mpls-oam-01). We expect to cover OAM for DetNet with these two OAM documents.
> The OAM for DetNet will be more complex than the OAM for a classical 
> P2P or P2MP LSP (or PW) because of the PREOF function. There is 
> nothing in the data plane that precludes us using on of the existing OAM indicators (GAL or 0001 ACH), but I think that it is important to thing through the subtleties. Thus I think it is OK to make progress on the elements of the data plane that we can nail down, and leave. The OAM as follow-up work.

> been published as RFC 8660 <Balazs>/<Stewart> OK, We will update.
> 

Thanks!

Carlos.

---

From: Lou Berger <[email-masked]>
Date: Mon, 27 Jan 2020 09:41:47 -0500

Hi,

Balázs thank you for the clarification -- see below for one comment.

>      zero S/N what simplifies implementation."

While I completely agree with the rational and validity of the good 
question, I don't think such motivation belongs in the document.  We 
generally don't document every design decision in a specification.  I 
don't feel strongly about this so if others do, I'll defer to their 
opinion...

Balázs, Carlos, Do you think it should stay?

Thanks,

Lou (as contributor)

---

From: "Carlos Pignataro (cpignata)" <[email-masked]>
Date: Mon, 27 Jan 2020 15:08:19 +0000

Hi, Lou,

> 2020/01/27 午前9:41、Lou Berger <[email-masked]>のメール:
> 
> Hi,
> 
> While I completely agree with the rational and validity of the good question, I don't think such motivation belongs in the document.  We generally don't document every design decision in a specification.  I don't feel strongly about this so if others do, I'll defer to their opinion...
> 

I have no strong feelings either way, but I believe if this is a departure from a “preferred” format from a BCP, then a one-liner explanation would not hurt. I’d leave this one in.

Thanks,

Carlos.