Subject: [Detnet] Some initial comments on draft-dt-detnet-dp-sol-00

From: Stewart Bryant <[email-masked]>
Date: Mon, 5 Jun 2017 21:01:31 +0100

I took an initial look at the detnet dp document.

I think that PWs are certainly a good way to go, but I am concerned
about a number of aspects of the proposal.

My biggest concern is doing the sequence number checking. There have
not been many attempts to do this, and from work that we have done in
the OAM space all the feedback I get is that this is a hard problem.
It might be useful if you constrained the packets to a path and an
interface using MPLS-TP or MPLS-TE, however I don't see a practical
way to do that in a pure IP network. You could of course do it
with SR, but that is v6 only, and the packets could potentially need
a very large SR header.

Please see the notes inline.

-Stewart

    The PW-based data plane can be run over either an IP
    or MPLS [RFC4448][RFC6658] Packet Switched Network (PSN).

SB> I comment on this in detail later

============

5.1.  DetNet Control Word

    The DetNet control word (d-CW) is identical to the control word
    defined for Ethernet over MPLS networks in [RFC4448].  The DetNet
    control word is illustrated in Figure 4.

       0                   1                   2                   3
       0 1 2 3 4 5 6 7 8 9 0 1 2 3 4 5 6 7 8 9 0 1 2 3 4 5 6 7 8 9 0 1
      +-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+
      |0 0 0 0|  reserved - set to 0  |   16 bit Sequence Number      |
      +-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+

                        Figure 4: DetNet Control Word

    [Editor's note: Shoudl we care about high speed links, here 16 bits
    of sequence number wraps fast?  For example, in a case of 100Gb/s
    link, 16 bits of sequence number will wrap in ~6.6ms assuming 1250
    octets of packets and ~3.3ms for 625 octets packets.  Both numbers
    mean quite long fiber distances, though.]

SB> This worries me. Doing read modify write on the sequence number is
SB> difficult in the general case, particularly without path constraint
SB> since a packet can arrive on any interface, and this interface can
SB> change.
SB>
SB> Are you thinking that there will be so few DN PWs that you can
SB> put the counters in registers? That might fly at the T-PEs, but
SB> I would be worried at the S-PEs.
SB>
SB> BTW shouldn't you consider much smaller packets, or do you imagine
SB> that DN will be constrained to applications using large packets.

===========

5.3.  DetNet encapsulation

    The DetNet data plane follows PW encapsulation.  This document
    specifies a single encapsulation that can be used over both MPLS and
    IP packet switched Networks (PSN).  The DetNet data plane
    encapsulation consists of a

    o  DetNet control word (d-CW): contains sequencing information for
       packet replication and duplicate elimination purposes.  There is a
       separate sequence number space per each DetNet label.

SB> Do you mean per DetNet flow ID, or per PW label?

  o  DetNet flow-ID (f-ID): uniquely identifies a DetNet flow within a
       DetNet network.  Multiple DetNet PWs with different PW labels may
       have the same f-ID, which then implies the PWs are actually
       subflows of one compound flow.

SB> I am not sure I understand the definition of a DetNet network yet.
SB> I presume that it is Fig 5 from the architecture draft, which
SB> is a tunnel between two service instances.
SB> However I am having difficulty understanding the scope of the
SB> uniqueness. It sounds as if it needs to be unique between a
SB> pair of service instances is that the case, or does the
SB> uniqueness have greater scope?
SB> PWs are not subflows, they are "A mechanism that carries the
SB> essential elements of an emulated service from one PE to one or
SB> more other PEs over a PSN."

  o  PseudoWire Label (PW Label;): a standard PW label that identifies
       a PW Instance within a (DA-)T-PE or (DA-)S-PE device.

SB> Just so we are all clear the PW label changes at S-PEs

===========

  o  DetNet topology overlay label (L-label): an optional label used
       between (DA-)T-PE or (DA-)S-PE nodes.  The main use of L-labels is
       to tunnel PWs through a PE node and therefore effectively making a
       PE node to behave like a P node.

SB> This needs more thought. The reason that S-PEs were created
SB> was to minimise the burden of running PWs between different
SB> administrative domains. To make this feasible it was necessary
SB> for the T-PEs to allocate their own PW label and have the
SB> S-PEs swap, that way only the boundary nodes (S-PEs) needed
SB> be worried about the mapping between the PE identity and the
SB> PW label in the data-plane.
SB> If a data-plane identifier is used, then we don't really need
SB> S-PEs as such. So I think that we have to define the new
SB> purpose of the S-PE more clearly when they are used for Detnet.

  In a case of MPLS-based PSN, the tunnel labels between LSRs are
    referred as T-labels.

SB> I think that they are really LSP labels.

  The DetNet CW and the Detnet flow-ID together constitute the DetNet
    PseudoWire encapsulation header.

       [Editor's note: The current design has the DetNet flow-ID as part
       of the every DetNet flow packet.  The flow-ID identifies the flow
       uniquely within the DetNet network and together with the sequence
       number information from the DetNet control word is used for PREF
       purposes.  The flow-ID makes is easy for the DA-*-PE node to
       associate different PWs into one compound flow and perform the
       elimination of duplicate packets.

SB> I am not sure this is needed. The DA-*-PE knows the relationship
SB> between the PW labels and can make the decision based on that.

       The flow-ID would point at the
       node internal construct that holds the received packet history for

Korhonen, et al.       Expires September 14, 2017               [Page 8]

Internet-Draft         DetNet data plane solution             March 2017

       each DetNet flow of interest.  However, it could also be possible
       to associate multiple PWs into one DetNet flow just using the
       control plane provided information.  In this case different PWs
       (using any PW label) would be mapped internally within a node to a
       local-ID (or similar construct), which again points at the
       internal per DetNet flow received packets history construct.

SB> I think you have to do this anyway

       The
       explicit in-band flow-ID is easy from the processing and control
       plane point of view.
SB> Isn't this a bigger change to the forwarder? What normally
SB> happens is you vector to the instructions and context using
SB> the PW label as the identifier. So I think the flow-ID just adds
SB> complexity. Given that it needs to be unique and known at
SB> each PW node on the path, I do not see what has been gained in
SB> terms of reduction in control plane activity.

       The local-ID approach does not need the in-
       band information (thus has less overhead) but requires more from
       the control plane and the mapping information has to be stored
       into the LFIB.  Current design decision is the in-band flow-ID but
       may be changed to local-ID if there is a strong reason to do the
       change.]

    Figure 6 illustrates a DetNet PseudoWire encapsulation using an MPLS
    PSN.  Similarly, Figure 7 illustrates the DetNet PseudoWire
    encapsulation when IP PSN is used.  The encapsulation is uniform
    above the PSN.

    Depending on the network topology the "overlay label" (L-label) may
    be part of the label stack.  The L-label tunnels guarantee PW labels
    remain unchanged between DA-*-PE nodes.
SB> Please see earlier

    Furthermore, L-labels
    tunnels allow selectively exposing the PW label to DA-*-PE nodes,
    which means some overlay topologies may just pass through specific
    DA-S-PEs without any DetNet specific processing.

SB> They can do this anyway. At an SPE we normally just swap the
SB> PW label, and other than for OAM reasons (trapped by TTL expiry)
SB> we do no processing. So if you do not teach an SPE that the PW
SB> is to be processed, it will just pass through.
SB>
SB> I think that you need to look seriously at deleting this component
SB> of your design and building something much closer to a normal
SB> PW design.

===============

    When IP PSN is used, the label stack it transports is only inspected
    when the IP packet destination address equals to the IP address of a
    DA-*-PE or a P node.  Essentially there are one more IP tunnels
    between a number of DA-*-PE and/or P nodes.  The LFIB and the
    forwarding information base (FIB) combination determines whether a PW
    gets terminated at the node or forwarded to another node within a new
    IP tunnel.

SB> So, setting aside for the moment the work that we are doing on
SB> unifying SRv6 and MPLS SR, work which as yet has no official status
SB> you should understand that there really is no deployment of
SB> MPLS-PW over IP. All of the deployed PWs are either pure PW over
SB> MPLS, or LT2Pv3. L2TPv3 has no concept of an S-PE.
SB>
SB> Now I think the unified approach is the right one, but so far
SB> there are no real specifications.
SB>
SB> Also you need to think about whether you want ECMP or not, because
SB> if you do you really do need the interstitial UDP layer (RFC7510)
SB> shown below.

==============

6.1.  Forwarded clarifications

    [Editor's note: The Detnet-aware "extended forwarder" does the heavy
    lifting on maintaining the sequence numbers associated with the
    DetNet labels.  Extended forwarder is also responsible for packet
    replication and duplicate elimination.  See the excerpt from RFC3985
    Section 4.2.1. about forwarder's functions.  We extend that to PREF:

       Some applications have to forward payload elements selectively
       from one or more ACs to one or more PWs.  In such cases, there
       will also be a need to perform the inverse function on PWE3-PDUs
       received by a PE from the PSN.  This is the function of the
       forwarder.

    ]
SB> But note that forwarder function only appears in the T-PE, we
SB> never included it in the S-PE which can be better thought of
SB> as a two layer MPLS switch - it's not how it works although
SB> a modern two label lookup MPLS system could do it like that
SB> but all that happens in an SPE is that you swap two labels
SB> simultaneously - the LSP label and the PW label.

     The DetNet specific new functionality in a DA-*-PE PW processing is
    the packet replication and duplication elimination function (PREF).
    This functional is a part of the "extended" forwarder.  The PREF
    processing is triggered by the LFIB actions i.e., not all PWs receive

Korhonen, et al.       Expires September 14, 2017              [Page 11]

Internet-Draft         DetNet data plane solution             March 2017

    DetNet specific processing.  Basically the LFIB has to be extended
    with a "PREF enabled" boolean configuration switch that is associated
    with the normal label actions (e.g., swap, push, pop, ..).  The
    output of the PREF elimination function is always a single packet.
    The output of the PREF replication function is always one or more
    packet (i.e., 1:M replication).  The replicated packets MUST share
    the same DetNet PW control word sequence number and flow identity
    word flow-id.

    The complex part of the DetNet PREF processing is tracking the
    history of received packets for multiple PWs.  These PWs do not have
    the same PW label value while they still share the same PW sequence
    number counter and the history information.  That is where the DetNet
    encapsulation header flow-ID plays an important role and binds the
    control word sequence number to the flow specific shared counter and
    history information within the PREF function.

SB> That is certainly one way of doing it, although given that you need
SB> to provision the PW anyway, I an not sure it is needed.

    The DetNet flow word contains a D flag bit (see Section 5.2), which
    makes the DA-*-PE node aware of the direction the flow-ID arrived
    from.  If the node, based on the local policy, checks for the D bit
    setting that effectively means the sequence number history has to
    contain also the D bit information.

SB> I am really not sure why you need the D bit a PW received on a given PW
SB> label only goes one way.

  ==============

7.  Other DetNet considerations

7.1.  Class of Service

    [Editor's note: Discuss the CoS.. and how that is archived when using
    MPLS or IP PSN.]

SB> Don't all your packets need to go on the highest class of service?

===========

7.3.  Time synchronization

    o  PTP with on-path support: in this approach PTP packets are sent as
       DetNet flows, and intermediate nodes take part in the protocol as
       Transparent Clocks or Boundary Clocks [IEEE1588].  The on-path PTP
       support by intermediate nodes provides a higher degree of accuracy
       than the previous approach.  The actual accuracy depends on
       whether all intermediate nodes are PTP-capable, or only a subset
       of them.

SB> RFC8169 shows how to do TC in an MPLS network.
SB> I am not sure anyone knows how to do this in an IP network.

===========

7.4.  Bidirectional traffic

    Some DetNet applications generate bidirectional traffic and may
    require symmetric flows.  There are already mechanisms that can be
    used to create bidirectional tunnels at the transport network level,
    such as MPLS-TP.  The data plane solution SHOULD allow establishing
    bidirectional symmetric flows.  Control plane mechanisms would need
    to also support this, though this is out of scope of this document.
    [Summary of existing mechanisms to create bidirectional tunnels that
    can be used.]

SB> PWs are always bidirectional of course.

  8.1.  PW Label assignment and distribution

    The PW label distribution follows the same mechanisms specified for
    MS-PW [RFC6073].

SB> This will need extensions to support DETNET

---

From: "Andrew G. Malis" <[email-masked]>
Date: Mon, 5 Jun 2017 16:55:28 -0400

In addition to Stewart’s comments, I had commented during the discussion of
this draft in Chicago that the PW definition aspects of the draft should be
split out to a separate draft that would belong in the PALS WG, taking
advantage of the PW expertise there.

Cheers,
Andy

On Mon, Jun 5, 2017 at 4:01 PM, Stewart Bryant <[email-masked]>
wrote:

> _______________________________________________
> detnet mailing list
> [email-masked]
> https://www.ietf.org/mailman/listinfo/detnet
>
>

---

From: Balázs Varga A <[email-masked]>
Date: Tue, 6 Jun 2017 16:50:39 +0000

Hi Stewart,

Thanks for the detailed review.

Based on several good comments in the Chicago meeting we have started to update the draft.
The major changes are:
1, changing encapsulation
- PW-based encapsulation: The PW-based data plane can be run over an MPLS PSN.
- Native-IP encapsulation: This solution is based on IP header fields, namely on IPv6 Flow label
and a new DetNet Control Word extension header option. It is targeted for native IPv6 networks.
2, PW-label specific requirement is simplified (regarding uniqueness):
- DetNet flows that need to undergo PREF processing MUST have the same PW Label when
they arrive at the DA-*-PE node.

We have also identified that sequence numbering related functions may be challenging, however
it is a must for the PREF (Packet Replication and Elimination Function) implementation. PREF provides
per packet level redundancy and not the so far defined per-PW-segment redundancy.

Similarly, your concerns on constraining DetNet packets to a path and an interface in a pure IP network,
is absolutely a candidate topic for further improvements.

I will be back soon with further feedback regarding your detailed notes.

Thanks & Cheers
Bala’zs

From: Stewart Bryant [mailto:[email-masked]]
Sent: 2017. június 5. 22:02
To: [email-masked]; [email-masked]
Subject: Some initial comments on draft-dt-detnet-dp-sol-00

a very large SR header.

-Stewart

============

5.1.  DetNet Control Word

      0                   1                   2                   3

SB> change.

SB>

SB>

===========

===========

   referred as T-labels.

      The

      plane point of view.

      change.]

   above the PSN.

SB> Please see earlier

   Furthermore, L-labels

SB>

SB> PW design.

===============

   IP tunnel.

SB>

SB>

SB> shown below.

==============

      forwarder.

   ]

   word flow-id.

 ==============

7.1.  Class of Service

   MPLS or IP PSN.]

===========

      of them.

===========

   can be used.]

   MS-PW [RFC6073].

---

From: Stewart Bryant <[email-masked]>
Date: Tue, 6 Jun 2017 18:12:23 +0100

> for native IPv6 networks.

It seems a shame that it needs to be different from the MPLS design. I 
was pointing out that
PW (MPLS format) over IP had not been deployed. That does not make it not a
nice design.

> they arrive at the DA-*-PE node.

That aligns with how PWs work.

> per packet level redundancy and not the so far defined per-PW-segment 

I think we need to draw up some use cases to understand what constraints 
we may need
to add in to make this at all feasible.

> is absolutely a candidate topic for further improvements.

The obvious solution is IP SR, but it needs to be a much lighterweight 
solution than the
solution we have on the table for V6 with one IPv6 address per specified 
hop along the
path.

Another way is to modify the routing protocol to constrain the path. We 
did that
in not-via, and could make constrain repair paths to avoid a failure, 
and could get
this to work in both link state and dv routing protocols. However we 
need the opposite.

Then I suppose RSVP was originally designed to do what you need. So I 
suppose
we could your RSVP to add explicit IP routing paths to the network.

Interesting problem.

> I will be back soon with further feedback regarding your detailed notes.

Well that was only a first pass, but it was somewhere to start.

BR

Stewart

> Thanks & Cheers
>
> Bala’zs
>
> *From:*Stewart Bryant [mailto:[email-masked]]
> *Sent:* 2017. június 5. 22:02
> *To:* [email-masked]; [email-masked]
> *Subject:* Some initial comments on draft-dt-detnet-dp-sol-00
>

---

From: Balázs Varga A <[email-masked]>
Date: Tue, 13 Jun 2017 16:06:47 +0000

Hi Stewart,
Further feedback on your detailed notes inline in your original mail.
Cheers
Bala’zs

From: Balázs Varga A
Sent: 2017. június 6. 18:51
To: 'Stewart Bryant' <[email-masked]>; [email-masked]; [email-masked]
Subject: RE: Some initial comments on draft-dt-detnet-dp-sol-00

Hi Stewart,

Thanks & Cheers
Bala’zs

From: Stewart Bryant [mailto:[email-masked]]
Sent: 2017. június 5. 22:02
To: [email-masked]<mailto:[email-masked]>; [email-masked]<mailto:[email-masked]>
Subject: Some initial comments on draft-dt-detnet-dp-sol-00

a very large SR header.

-Stewart

============

5.1.  DetNet Control Word

      0                   1                   2                   3

SB> change.

SB>

SB>

[Balázs Varga A]

Yes, Doing read modify write on the sequence number is difficult in the general case.

However here we do write only at ingress node (e.g., T-PE). No modify operation defined.

Read operation is needed on S-PE nodes, doing replication and elimination and they need

a counter per flow. That is the tax of per packet redundancy. Not all flow may require PREF

function of DetNet. Resources have to be considered during flow setup.

===========

[Balázs Varga A]

In case of MPLS DetNet flow ID = PW label.

[Balázs Varga A]

the scope of the uniqueness is that it needs to be unique between a pair of service instances.

[Balázs Varga A]

In DetNet, PWs are used to transport the subflows (replica flows) of a compound flow. Inline

with your quoted definition.

[Balázs Varga A]

Yes, the PW label may change.

===========

[Balázs Varga A]

There is dedicated chapter in the new data-plane draft version to describe in more detail S-PE

nodes. I think that will be a good starting point for further clarification.

   referred as T-labels.

[Balázs Varga A]

Yes, they are LSP labels.

[Balázs Varga A]

That is a point where the design team had a long discussion. Table lookups were calculated and

resulted in this note. I agree it may be implementation specific, so further details would be great

to discuss.

      The

      plane point of view.

[Balázs Varga A]

That would be worth for further discussions with chip vendors onboard.

      change.]

   above the PSN.

SB> Please see earlier

   Furthermore, L-labels

SB>

SB> PW design.

[Balázs Varga A]

This component is optional. There are some scenarios when it can be useful.

E.g., on Flow identification for OAM purposes. We have discussed some troubleshooting scenario,

where it was necessary on a link to being able to distinguish which DetNet packet belongs to which

sub-flow of a compound flow.

Example:

sub-flow1: A --> D

sub-flow2: B --> D

So on link C-D we can NOT distinguish packets of the two sub-flow, as they have the same PW-label.

A--+

       \

        C-----D

       /

B--+

One possible solution is the use of the "Topology overlay Label".

===============

   IP tunnel.

SB>

SB>

SB> shown below.

[Balázs Varga A]

This is fixed now in the new version of the draft. See my previous mail.

==============

      forwarder.

   ]

   word flow-id.

[Balázs Varga A]

This is fixed in the new version. PWs of the same compound flow have the same PW label.

[Balázs Varga A]

If I remember correctly the scenario was for rings with clockwise and anticlockwise directions.

 ==============

7.1.  Class of Service

   MPLS or IP PSN.]

[Balázs Varga A]

It is scenario and network operator specific. One may decide that some OAM traffic may have higher

precedence than anything else.

===========

      of them.

[Balázs Varga A]

I think for IP hop-by-hop PTP support would be preferred.

===========

   can be used.]

[Balázs Varga A]

Yes.

   MS-PW [RFC6073].

[Balázs Varga A]

Yes.