Subject: [Detnet] WG adoption poll draft-dt-detnet-dp-sol

From: Pat Thaler <[email-masked]>
Date: Tue, 10 Oct 2017 10:27:30 -0700

All,

This is start of a two week poll on making
draft-dt-detnet-dp-sol a working group document. Please
send email to the list indicating "yes/support" or "no/do not
support".  If indicating no, please state your reservations with the
document.  If yes, please also feel free to provide comments you'd
like to see addressed once the document is a WG document.

Please remember that working group adoption is the start of the
WG's work on the draft, not the end of it. This is not a typical
adoption call in so far as that the document has captured open
issues that have previously been raised by the WG and are
expected to be resolved through normal WG process as the
document progresses.

The poll ends October 24

Thanks,

Pat and Lou,

---

From: Balázs Varga A <[email-masked]>
Date: Tue, 10 Oct 2017 17:33:09 +0000

Yes, I support workgroup adaption.
Cheers Bala’zs

From: detnet [mailto:[email-masked]] On Behalf Of Pat Thaler
Sent: 2017. október 10. 19:28
To: [email-masked]
Subject: [Detnet] WG adoption poll draft-dt-detnet-dp-sol

All,

The poll ends October 24

Thanks,

Pat and Lou,

---

From: Alon Regev <[email-masked]>
Date: Tue, 10 Oct 2017 17:37:33 +0000

yes/support

Regards,
Alon

From: detnet [mailto:[email-masked]] On Behalf Of Pat Thaler
Sent: Tuesday, October 10, 2017 10:28 AM
To: [email-masked]
Subject: [Detnet] WG adoption poll draft-dt-detnet-dp-sol

All,

The poll ends October 24

Thanks,

Pat and Lou,

---

From: Henrik Austad <[email-masked]>
Date: Tue, 10 Oct 2017 19:49:25 +0200

Yes, I support workgroup adoption.

-Henrik

On Tue, Oct 10, 2017 at 7:27 PM, Pat Thaler <[email-masked]> wrote:
> _______________________________________________
> detnet mailing list
> [email-masked]
> https://www.ietf.org/mailman/listinfo/detnet
>

-- 
Henrik Austad

---

From: "Grossman, Ethan A." <[email-masked]>
Date: Tue, 10 Oct 2017 17:56:39 +0000

?yes/support

Ethan.

________________________________
From: detnet <[email-masked]> on behalf of Pat Thaler <[email-masked]>
Sent: Tuesday, October 10, 2017 10:27 AM
To: [email-masked]
Subject: [Detnet] WG adoption poll draft-dt-detnet-dp-sol

All,

The poll ends October 24

Thanks,

Pat and Lou,

---

From: Norman Finn <[email-masked]>
Date: Tue, 10 Oct 2017 18:04:51 +0000

yes/support

-- Norm

________________________________
From: detnet [[email-masked]] on behalf of Pat Thaler [[email-masked]]
Sent: Tuesday, October 10, 2017 10:27 AM
To: [email-masked]
Subject: [Detnet] WG adoption poll draft-dt-detnet-dp-sol

All,

The poll ends October 24

Thanks,

Pat and Lou,

---

From: Stewart Bryant <[email-masked]>
Date: Tue, 10 Oct 2017 19:14:35 +0100

I would prefer that we resolved some of the fundamental issues before 
adoption, and do not see why we are adopting the process proposed.

I am not sure the WGC have properly explained the need to rush to the 
point of a WG draft, and think that this needs to be articulated.

That said, if we really must adopt the draft at this stage, I am pleased 
to see that it has the issues clearly marked, and look forward to their 
resolution.

- Stewart

On 10/10/2017 18:27, Pat Thaler wrote:
> draft-dt-detnet-dp-sol a workinggroup document. Please

---

From: David Sinicrope <[email-masked]>
Date: Tue, 10 Oct 2017 18:28:25 +0000

I support working group adoption of this draft.
While the PW solution is key, I also think it is important to consider an EVPN-based solution.
Dave

From: detnet <[email-masked]> on behalf of Pat Thaler <[email-masked]>
Date: Tuesday, October 10, 2017 at 1:29 PM
To: "[email-masked]" <[email-masked]>
Subject: [Detnet] WG adoption poll draft-dt-detnet-dp-sol

All,

The poll ends October 24

Thanks,

Pat and Lou,

---

From: "Andrew G. Malis" <[email-masked]>
Date: Tue, 10 Oct 2017 16:02:40 -0400

Pat,

Like Stewart, I would like to see resolution of the open issues identified
in the current revision of the draft, but I don’t see any reason why that
can’t be done as a WG draft. Driving the resolution process should be first
thing done by the appointed primary editor.

Cheers,
Andy

---

From: Jiangyuanlong <[email-masked]>
Date: Wed, 11 Oct 2017 06:46:14 +0000

Yes/Support.

Yuanlong

From: detnet [mailto:[email-masked]] On Behalf Of Pat Thaler
Sent: Wednesday, October 11, 2017 1:28 AM
To: [email-masked]
Subject: [Detnet] WG adoption poll draft-dt-detnet-dp-sol

All,

The poll ends October 24

Thanks,

Pat and Lou,

---

From: "Pascal Thubert (pthubert)" <[email-masked]>
Date: Wed, 11 Oct 2017 07:17:30 +0000

yes/support

From: detnet [mailto:[email-masked]] On Behalf Of Pat Thaler
Sent: mardi 10 octobre 2017 19:28
To: [email-masked]
Subject: [Detnet] WG adoption poll draft-dt-detnet-dp-sol

All,

The poll ends October 24

Thanks,

Pat and Lou,

---

From: Rodney Cummings <[email-masked]>
Date: Wed, 11 Oct 2017 14:11:30 +0000

yes/support

From: detnet [mailto:[email-masked]] On Behalf Of Pat Thaler
Sent: Tuesday, October 10, 2017 12:28 PM
To: [email-masked]
Subject: [Detnet] WG adoption poll draft-dt-detnet-dp-sol

All,

The poll ends October 24

Thanks,

Pat and Lou,

---

From: "Patrick Wetterwald (pwetterw)" <[email-masked]>
Date: Thu, 12 Oct 2017 15:43:46 +0000

Yes/support.

Patrick

From: detnet <[email-masked]> on behalf of Pat Thaler <[email-masked]>
Date: Tuesday, 10 October 2017 at 19:27
To: "[email-masked]" <[email-masked]>
Subject: [Detnet] WG adoption poll draft-dt-detnet-dp-sol

All,

The poll ends October 24

Thanks,

Pat and Lou,

---

From: Jouni <[email-masked]>
Date: Mon, 16 Oct 2017 23:19:52 +0300

Yes/support.

> On 10 Oct 2017, at 20:27, Pat Thaler <[email-masked]> wrote:
> 
> All,
>

---

From: CARLOS JESUS BERNARDOS CANO <[email-masked]>
Date: Mon, 16 Oct 2017 23:03:11 +0200

Yes/support.

Carlos

---

From: Loa Andersson <[email-masked]>
Date: Wed, 18 Oct 2017 14:05:20 +0200

Working Group, co-authors,

After talking to a number or people I've re-reviewed that the Packet
Replication and  Elimination Function (PREF). It seems that there is a
strong relationship between router architecture and the possibility to
fully perform this function. Replication is fine with any architecture.

However, if you have an architecture with multiple line cards and with
packet processing on the ingress, performing elimination seems to be 
very hard, and would require major changes to the routers.

As long as you have a one chip router both replication and elimination
is possible.

Since a substantial part of all routers that would be in scope for
DetNet networks are router with multiple line cards where the packet
processing is done on the ingress cards, the elimination function seems
to be a major problem.

We have said that we can do elimination on the out port, but it seems
that most routers do not have the required packet processing on the
egress line cards.

/Loa

PS

Please note that I don't think this is a comment blocking the adoption
as a wg doc.

On 2017-10-11 16:11, Rodney Cummings wrote:
> yes/support
> 
> *From:*detnet [mailto:[email-masked]] *On Behalf Of *Pat Thaler
> *Sent:* Tuesday, October 10, 2017 12:28 PM
> *To:* [email-masked]
> *Subject:* [Detnet] WG adoption poll draft-dt-detnet-dp-sol
> 
> All,
> 

-- 

Loa Andersson                        email: [email-masked]
Senior MPLS Expert
Huawei Technologies (consultant)     phone: +46 739 81 21 64

---

From: "Pascal Thubert (pthubert)" <[email-masked]>
Date: Wed, 18 Oct 2017 12:45:12 +0000

Hi Loa:

Arguably, elimination and reordering belong to an upper layer - like a layer 4-, that can be run either at the detnet end systems, or at very specific locations in the network where an excursion to L-4  and back down is permitted. We had text that showed that replication could be done at a lower layer, but that must have been edited out the architecture, I could not dig it. I'm full speed preparing a draft on that matter (to be sent to TSVWG), hopefully I'll push a 00 early next week.

All the best,

Pascal

-----Original Message-----
From: detnet [mailto:[email-masked]] On Behalf Of Loa Andersson
Sent: mercredi 18 octobre 2017 14:05
To: Pat Thaler <[email-masked]>; [email-masked]
Subject: Re: [Detnet] WG adoption poll draft-dt-detnet-dp-sol

After talking to a number or people I've re-reviewed that the Packet Replication and  Elimination Function (PREF). It seems that there is a strong relationship between router architecture and the possibility to fully perform this function. Replication is fine with any architecture.

However, if you have an architecture with multiple line cards and with packet processing on the ingress, performing elimination seems to be very hard, and would require major changes to the routers.

As long as you have a one chip router both replication and elimination is possible.

Since a substantial part of all routers that would be in scope for DetNet networks are router with multiple line cards where the packet processing is done on the ingress cards, the elimination function seems to be a major problem.

We have said that we can do elimination on the out port, but it seems that most routers do not have the required packet processing on the egress line cards.

/Loa

PS

Please note that I don't think this is a comment blocking the adoption as a wg doc.

On 2017-10-11 16:11, Rodney Cummings wrote:
> This is start of a two week poll on making draft-dt-detnet-dp-sol a 
> working group document. Please send email to the list indicating 
> "yes/support" or "no/do not support".  If indicating no, please state 
> your reservations with the document.  If yes, please also feel free to 
> provide comments you'd like to see addressed once the document is a WG 
> document.
> 

--

---

From: Loa Andersson <[email-masked]>
Date: Wed, 18 Oct 2017 21:46:50 +0800

Pasca),

Are saying that in a DetNet network we’ll kick ALL packets into the slow path in all routers that do elimination?

/Loa

Sent from my iPhone

> On 18 Oct 2017, at 20:45, Pascal Thubert (pthubert) <[email-masked]> wrote:
> 
> Hi Loa:
>

---

From: "Pascal Thubert (pthubert)" <[email-masked]>
Date: Wed, 18 Oct 2017 14:03:14 +0000

I do not think I said that Loa. 

For one thing, slow path is an implementation thing and I was discussing layers. And then what is put in slow path is another implementation decision. 

Rather I said that the function belongs to a layer that should not be expected in an intermediate router. IOW a basic DetNet  Router would be implementing the DetNet Transport Layer (which is not a Layer 4, confusing...) and would not need to reorder or eliminate.
 
I'm looking at how the DetNet Service Layer is positioned vs. a traditional transport layer, considering how the architecture can alleviate the issue that you are raising. The architecture is more open than my words here, but I think we need to restrict this function to DetNet End Systems and maybe Edge Routers that have a special excursion to L4, like a transport layer proxy implementing a RED box. 

Does that make sense?

Pascal

-----Original Message-----
From: Loa Andersson [mailto:[email-masked]] 
Sent: mercredi 18 octobre 2017 15:47
To: Pascal Thubert (pthubert) <[email-masked]>
Cc: Pat Thaler <[email-masked]>; [email-masked]
Subject: Re: [Detnet] WG adoption poll draft-dt-detnet-dp-sol

Pasca),

/Loa

Sent from my iPhone

>> your reservations with the document.  If yes, please also feel free 
>> to provide comments you'd like to see addressed once the document is 
>> a WG document.
>>

---

From: Loa Andersson <[email-masked]>
Date: Thu, 19 Oct 2017 14:58:17 +0800

Pascal ,

Sent from my iPhone

> On 18 Oct 2017, at 22:03, Pascal Thubert (pthubert) <[email-masked]> wrote:
> 

I’m not sure. Both the architecture (e.g. Figure 1) and the problem statement (section 3.5) clearly indicate that both replication and elimination are done “at several points”, including in the core. 

If we limit replication to the ingress and elimination to the egress this is a simple 1+1 protection. I thought replication/elimination points were introduced because 1+1 were not considered efficient enough.

Also if we put all elimination in the egress “in a layer 4 function” we still don’t get around the problem. A egress node doing packet processing on multiple line cards have to send ALL packets to the “layer 4 function”. 

/Loa

---

From: "Pascal Thubert (pthubert)" <[email-masked]>
Date: Thu, 19 Oct 2017 10:04:16 +0000

I agree Loa, this is where I stand too.

The traffic from the line cards must converge to a point where enough buffers are available to cope with the difference of latency over the various paths. The end-to-end latency may suffer from having it. OTOH having a point like that inside the network is necessary for Packet Replication and Elimination, even more when wireless joins the party. It is also necessary to redistribute a DetNet flow over one or more outgoing TSN  "circuits"  that have different capabilities than the incoming ones. 

How that point is implemented may differ from a product to the next, but that cannot always be done in line cards, for the reasons you point out, but also for reasons of buffer capacity, mostly for the reordering feature.

Cheers,

Pascal

-----Original Message-----
From: Loa Andersson [mailto:[email-masked]] 
Sent: jeudi 19 octobre 2017 08:58
To: Pascal Thubert (pthubert) <[email-masked]>
Cc: Pat Thaler <[email-masked]>; [email-masked]
Subject: Re: [Detnet] WG adoption poll draft-dt-detnet-dp-sol

Pascal ,

Sent from my iPhone

/Loa
>>> "yes/support" or "no/do not support".  If indicating no, please 
>>> state your reservations with the document.  If yes, please also feel 
>>> free to provide comments you'd like to see addressed once the 
>>> document is a WG document.
>>>

---

From: "Andrew G. Malis" <[email-masked]>
Date: Thu, 19 Oct 2017 11:18:46 -0400

Pascal,

Yes, end-to-end latency may suffer if in-order delivery reliability is more
important, so that you buffer out-of-order packets until previous packets
have arrived and can be delivered. There are various ways things can be
tweaked depending on the relative importance of delivery and latency; if
latency is more important than guaranteed delivery, then out-of-order
packets can be dropped if they arrive after a later-in-the-sequence packet
has already been delivered, and a simple “last packet delivered” counter
per flow can also be used for duplicate elimination. As has been noted,
this has to be done at a common processing point, unless multiple ingress
line cards have shared memory for that counter.

Cheers,
Andy

On Thu, Oct 19, 2017 at 6:04 AM, Pascal Thubert (pthubert) <
[email-masked]> wrote:

> The traffic from the line cards must converge to a point where enough
> buffers are available to cope with the difference of latency over the
> various paths. The end-to-end latency may suffer from having it. OTOH
> having a point like that inside the network is necessary for Packet
> Replication and Elimination, even more when wireless joins the party. It is
> also necessary to redistribute a DetNet flow over one or more outgoing TSN
> "circuits"  that have different capabilities than the incoming ones.
>
> How that point is implemented may differ from a product to the next, but
> that cannot always be done in line cards, for the reasons you point out,
> but also for reasons of buffer capacity, mostly for the reordering feature.
>
> Cheers,
>
> Pascal
>
>
> > For one thing, slow path is an implementation thing and I was discussing
> layers. And then what is put in slow path is another implementation
> decision.
> >
> > Rather I said that the function belongs to a layer that should not be
> expected in an intermediate router. IOW a basic DetNet  Router would be
> implementing the DetNet Transport Layer (which is not a Layer 4,
> confusing...) and would not need to reorder or eliminate.
> >
> > I'm looking at how the DetNet Service Layer is positioned vs. a
> traditional transport layer, considering how the architecture can alleviate
> the issue that you are raising. The architecture is more open than my words
> here, but I think we need to restrict this function to DetNet End Systems
> and maybe Edge Routers that have a special excursion to L4, like a
> transport layer proxy implementing a RED box.
> >
> > Does that make sense?
>
> I’m not sure. Both the architecture (e.g. Figure 1) and the problem
> statement (section 3.5) clearly indicate that both replication and
> elimination are done “at several points”, including in the core.
>
> If we limit replication to the ingress and elimination to the egress this
> is a simple 1+1 protection. I thought replication/elimination points were
> introduced because 1+1 were not considered efficient enough.
>
> Also if we put all elimination in the egress “in a layer 4 function” we
> still don’t get around the problem. A egress node doing packet processing
> on multiple line cards have to send ALL packets to the “layer 4 function”.
>
> /Loa
> >
> > Pascal
> >
> >> Arguably, elimination and reordering belong to an upper layer - like a
> layer 4-, that can be run either at the detnet end systems, or at very
> specific locations in the network where an excursion to L-4  and back down
> is permitted. We had text that showed that replication could be done at a
> lower layer, but that must have been edited out the architecture, I could
> not dig it. I'm full speed preparing a draft on that matter (to be sent to
> TSVWG), hopefully I'll push a 00 early next week.
> >>
> >> All the best,
> >>
> >> Pascal
> >>
> >>
> >>
> packet processing on the ingress, performing elimination seems to be very
> hard, and would require major changes to the routers.
> >>
> processing is done on the ingress cards, the elimination function seems to
> be a major problem.
> >>
> that most routers do not have the required packet processing on the egress
> line cards.
> >>
> >> /Loa
> >>
> >> PS
> >>

---

From: "Jouni" <[email-masked]>
Date: Tue, 24 Oct 2017 12:23:45 +0300

Hi Loa,

> Sent: Wednesday, October 18, 2017 15:05 PM

This was our lowest hanging fruit conclusion during the numerous DT discussions around this topic.

> Since a substantial part of all routers that would be in scope for DetNet
> networks are router with multiple line cards where the packet processing
> is done on the ingress cards, the elimination function seems to be a major
> problem.
> 
> We have said that we can do elimination on the out port, but it seems that
> most routers do not have the required packet processing on the egress line
> cards.

I would say that like some other assumptions we have within DetNet we should understand that certain features are likely to be challenging to the existing design/silicon/software and we cannot get stuck into that. How big the delta will be to existing is IMHO important.

- Jouni

---

From: Stewart Bryant <[email-masked]>
Date: Tue, 24 Oct 2017 11:05:18 +0100

Unfortunately I don't think that it is quite as simple as requiring 
single chip implementations. You have to look at the internal design of 
that chip and the performance requirements.

Elimination needs to be an atomic operation in which the sequence number 
is locked, read, compared with the packet, modified and written back to 
the Detnet parameter set for that detwire. There are two difficulties to 
be considered here, firstly that it is not uncommon for Network 
Processors to be multi-core and secondly if there are a significant 
number of detwires, as there might be passing through a elim/rep node in 
the core of the network, it likely that the parameter set will be held 
away from the processing cores, possibly in memory requiring a DMA to 
fetch the data. Thus doing the elimination in these architectures will 
stall the pipeline and represent a big performance hit even in a single 
chip router.

Now let's go up a level. What is being proposed here is a major change 
in IP/MPLS forwarding. Pseudowires, on which you model the detnet 
design, did not make any such change. The operation in the S-PE was 
identical the common MPLS operation of label swap and forward to a 
tunnel. Something that can be compressed into a single re-write 
operation all the way down to the MAC imposition. I am deeply concerned 
that my making detnet look like PWs people will assume that 
implementation will be as simple and effective as PWs. Thus I think the 
current description is misleading.

In mainstream forwarding with two exceptions we have a design that is 
fully asynchronous allowing any forwarding component to proceed without 
the need to liaise with any other component in the forwarding system. 
The two exceptions are counters, which look like RMW, but there are 
pipeline tricks that can be applied such that RMW is not actually done. 
The other is packet ordering. Packet ordering is only best effort and is 
not maintained through convergence. The reason we do this to help TCP, 
but the failure is only a soft failure (some TCP implementations may 
close the window but they recover). On the other hand we are told that 
some detnet applications take a fatal error if there is ever a misorder. 
Avoiding any misorder under any circumstances required analysis of the 
routing layer behaviour that I have not yet seen reported in this project.

There may be internal tricks that can be played in the internal design 
of forwarders to make this work, even for multi-chassis designs, but the 
cost of this is the removal of a number of router architectures from the 
set from which the forwarder designer can choose, and to me that is a 
big decision that requires significant discussion amongst those with 
appropriate expertise. Unfortunately few of those engineers actively 
engage with the IETF because it is so long since we last changed an 
IP/MPLS invariant of this magnitude.

My view is that we need a serious discussion in the IETF about what 
invariants we need to maintain in the forwarding path across the core. 
Wherever the consensus of that discussion is, needs to be reflected 
carefully and fully in this document if it is to go ahead.

What I think might be a better approach is to eliminate elimination from 
this document and put it in a separate RFC. That way it can be made much 
clearer to operators what the consequences of elimination are and 
whether it is supported by a particular product.

- Stewart

---

From: Norman Finn <[email-masked]>
Date: Tue, 24 Oct 2017 18:54:53 +0000

Several good points have been made by several people.  I'll add and/or reinforce some, and contest some others:

Splitting out replication/elimination:

Any router can punt replication/elimination to software, and do a great job on replication/elimination, but be useless from the latency standpoint.  I have developed and shipped an awful lot of federated devices (line cards + mid- or back-plane + controller) that would see no significant hit on performance when executing packet replication/elimination, though they would have tight resource constraints on the number of flows supported.  I have shipped quite a number of devices that could do a great job on replication/elimination, but only if you use a chunk of an FPGA facility.  That might mean sacrificing some other feature, presumably one not needed in this environment.  I have also been associated with devices that are cost-engineered and optimized for a function so tightly that they can accommodate no such feature as this.  The simple statement, "packet replication/elimination requires a one-chip router" is much too simple to be useful, and I would object to inserting it.

The "average" one-chip device is likely to find replication/elimination easier than the "average" federated device.  That means nothing.  What matters is which of *my* devices can implement this feature.  IMO it's a bad idea to split the document in two based on, "some implementations may find it difficult."  Lists of reasons why it may or may not be difficult (INCLUDING MINE!) are pointless, though fun to write.

Now, if more than one vendor steps in and says, "I want this split out, because my devices can't do it efficiently," I'm ready to listen and change things.  In the meantime, I think it's a good idea to put something in the document to the effect of, "before you get all enthusiastic about this feature, please look at this potential implementation impediment, and see whether your devices will find it difficult."

I would like to see more mention in the DetNet data plane drafts of flows that do not require replication/elimination, in order to make it clear that DetNet does not equal rep/elim.  (I think they're too focused on that one aspect, because, of course, it is the tough one to handle.)

But, the point of DetNet is twofold: 1) bounded worst-case latency; and 2) extraordinarily low packet loss.  So, I do want all the main elements in the architecture document.

In-order delivery:

The architecture draft does not say much about in-order delivery.  802.1CB (frame replication and elimination) makes it a specific non-goal, though it gives you clues about how it could be done.   The use cases draft states explicitly that it is not interested in in-order delivery for video streams, because it's too hard.  The difficulty of in-order delivery has been discussed a lot, both in IEEE and in DetNet, an the conclusion was, "forget about it."

Delivery order is therefore not an argument against replication/elimination.

In general, routers do not offer guarantees of in-order delivery, except insofar as sequence numbering in various forms provides an indication that mis-ordering has happened, and an opportunity to correct it.  Some applications find that useful.  DetNet changes nothing in that regard.

I agree, however, that there may be a blind spot in the drafts with this regard (generated by the authors' over-familiarity with the subject).  Some mention of in-order delivery might be useful, along the following lines:  A. For many DetNet applications, the transmission interval for any given flow is longer than the delivery time, so the opportunity for out-of-order delivery is strictly limited, and can be engineered out. B. Given point A, the use of dual fixed paths and rep/elim, along with worst-case latency guarantees, provides useful limits on out-of-order delivery, even when the paths are much extended by DetNet.  Whether this belongs in architecture, dp-sol, or some other draft is a good question.

-- Norm
 
________________________________________
From: detnet [[email-masked]] on behalf of Stewart Bryant [[email-masked]]
Sent: Tuesday, October 24, 2017 3:05 AM
To: Jouni; 'Loa Andersson'; 'Pat Thaler'; [email-masked]
Subject: Re: [Detnet] WG adoption poll draft-dt-detnet-dp-sol

- Stewart

---

From: Lou Berger <[email-masked]>
Date: Tue, 24 Oct 2017 15:26:45 -0400

I want to pick up on one point made below:

> Pseudowires, on which you model the detnet design, ...

I think this touches on one major open issue we (as a WG) should close
on sooner rather than later, specifically do we based DetNet over MPLS
on PWs or on EVPN.  (This is Comment #3.1, which is credited to me but
was actually raised at IETF 99, by Jeff T if my memory serves.)

Lou

---

From: "Jouni" <[email-masked]>
Date: Tue, 24 Oct 2017 22:33:26 +0300

Stewart,

> From: Stewart Bryant [mailto:[email-masked]]
> Sent: Tuesday, October 24, 2017 13:05 PM
> To: Jouni <[email-masked]>; 'Loa Andersson' <[email-masked]>; 'Pat
> Thaler' <[email-masked]>; [email-masked]
> Unfortunately I don't think that it is quite as simple as requiring single
> chip implementations. You have to look at the internal design of that chip
> and the performance requirements.

No one was/is _requiring_ single chip implementations. It was just concluded to be the easiest solution to the problem. We did not find a good solution but did not want to get stuck to it either. All input is welcome.

[snip]

> Now let's go up a level. What is being proposed here is a major change in
> IP/MPLS forwarding. Pseudowires, on which you model the detnet design, did
> not make any such change. The operation in the S-PE was identical the
> common MPLS operation of label swap and forward to a tunnel. Something
> that can be compressed into a single re-write operation all the way down
> to the MAC imposition. I am deeply concerned that my making detnet look
> like PWs people will assume that implementation will be as simple and
> effective as PWs. Thus I think the current description is misleading.

Ok, a fair statement.
 
[snip]

> window but they recover). On the other hand we are told that some detnet
> applications take a fatal error if there is ever a misorder.

If one is to emulate e.g. a serial link what else you can do than deliver packets to the end system in-order. However, the solution proposal did not say anything about it because the architecture document pushed the hardship to end systems.

[snap]

> clearer to operators what the consequences of elimination are and whether
> it is supported by a particular product.

This is something to think about. I would not rush to it as the first step though.

- Jouni

> >> Replication and  Elimination Function (PREF). It seems that there is
> >> a strong relationship between router architecture and the possibility
> >> to fully perform this function. Replication is fine with any
> architecture.
> >>
> >> However, if you have an architecture with multiple line cards and
> >> with packet processing on the ingress, performing elimination seems
> >> to be very hard, and would require major changes to the routers.
> >>
> >> processing is done on the ingress cards, the elimination function
> >> seems to be a major problem.
> >>
> > I would say that like some other assumptions we have within DetNet we
> should understand that certain features are likely to be challenging to
> the existing design/silicon/software and we cannot get stuck into that.
> How big the delta will be to existing is IMHO important.
> >
> > - Jouni
> >
> >
> >
> >> /Loa
> >>
> >> PS
> >>

---

From: Pat Thaler <[email-masked]>
Date: Wed, 25 Oct 2017 11:27:42 -0700

All,

    This poll is complete (as of Oct 25).  The document is adopted.
Clearly, the draft provides a starting point but there is significant work
ahead on it for the Working Group.

Authors,

    Please republish the draft-dt-detnet-dp-sol-02 as
draft-ietf-detnet-dp-sol-00 with the filename and date being the only
changes.

Thank you!

Pat and (Lou)

On Tue, Oct 24, 2017 at 12:33 PM, Jouni <[email-masked]> wrote:

> single
> No one was/is _requiring_ single chip implementations. It was just
> concluded to be the easiest solution to the problem. We did not find a good
> solution but did not want to get stuck to it either. All input is welcome.
>
> [snip]
>
> If one is to emulate e.g. a serial link what else you can do than deliver
> packets to the end system in-order. However, the solution proposal did not
> say anything about it because the architecture document pushed the hardship
> to end systems.
>
> [snap]
>

---

From: Norman Finn <[email-masked]>
Date: Wed, 25 Oct 2017 20:35:13 +0000

Layering is clearly spelled out in the architecture.  Replication/elimination is in the DetNet Service layer,
and the latency/congestion pieces are in the DetNet Transport layer below it.  Those are real layers.
They are described as such, and can be readily identified as such in the packet formats by the extra
MPLS and pseudowire-ish labels or the added 802.1 or HSR/PRP tags.  (Don't let the IPv6 format
fool you into thinking we don't have two layers, here.)  They can also be stacked up; there is no
fundamental reason that an end-to-end rep/elim compound flow can't have one of its member flows
split and merged at a lower layer, because the network has some flaky links along its path.  Yes:
pswudowires inside MPLS labels inside pseudowires inside MPLS labels.

Rep/elim can be done at higher layers, already.  But, with the current positioning and the packet formats
chosen, there is a good chance that the job can be done in ASICs or FPGAs very close to where things
are done, now.  Replication/elimination is somewhat more complex than, but takes place at
exactly the same layer as, pseudowire stitching.  I'm confident that, if the users want it, it can be
implemented at a reasonable cost; that's what FPGA sockets are for.

-- Norm

________________________________________
From: detnet [[email-masked]] on behalf of Loa Andersson [[email-masked]]
Sent: Wednesday, October 18, 2017 11:58 PM
To: Pascal Thubert (pthubert)
Cc: [email-masked]; Pat Thaler
Subject: Re: [Detnet] WG adoption poll draft-dt-detnet-dp-sol

Pascal ,

Sent from my iPhone

/Loa

---

From: Stewart Bryant <[email-masked]>
Date: Thu, 26 Oct 2017 11:50:10 +0100

On 24/10/2017 19:54, Norman Finn wrote:
> Any router can punt replication/elimination to software, and do a great job on replication/elimination, but be useless from the latency standpoint.
I agree.

>   I have developed and shipped an awful lot of federated devices (line cards + mid- or back-plane + controller) that would see no significant hit on performance when executing packet replication/elimination, though they would have tight resource constraints on the number of flows supported.
In the core there is a real risk that this would be tight to the point 
of limited use and having huge operational and commercial difficulties. 
For example having to create sub optimal paths to work around capacity 
limits and having to deal with rationing such resources amongst 
competing customers.

> I have shipped quite a number of devices that could do a great job on replication/elimination, but only if you use a chunk of an FPGA facility.  That might mean sacrificing some other feature, presumably one not needed in this environment.
This is not so easy in the WAN.
> I have also been associated with devices that are cost-engineered and optimized for a function so tightly that they can accommodate no such feature as this.  The simple statement, "packet replication/elimination requires a one-chip router" is much too simple to be useful, and I would object to inserting it.
I am not sure that a one chip constraint is adequate. Equally I think 
there are processing architectures in federated architectures that can 
be made to work. The issue is the cost in hardware, the cost in 
performance and the cost in operational and commercial complexity.

> The "average" one-chip device is likely to find replication/elimination easier than the "average" federated device.  That means nothing.  What matters is which of *my* devices can implement this fe…

I am afraid I disagree. We should disclose why this is a much harder 
problem than it appears on the surface.

> Now, if more than one vendor steps in and says, "I want this split out, because my devices can't do it efficiently," I'm ready to listen and change things.  In the meantime, I think it's a good ide…

I think that we need more detail that than that in the text.

> I would like to see more mention in the DetNet data plane drafts of flows that do not require replication/elimination, in order to make it clear that DetNet does not equal rep/elim.  (I think they'…

I do agree with that.

> Delivery order is therefore not an argument against replication/elimination.

Ah in that case we really need text on that.

It would also be useful to have text on the issue of whether elimination 
can be best effort or not.

My comments have been made on the basis that complete elimination of 
duplicates was required. If we relax that constraint then the 
implementation becomes easier, although not without a significant hit in 
terms of number of flows or performance in many implementation 
architectures.

> In general, routers do not offer guarantees of in-order delivery, except insofar as sequence numbering in various forms provides an indication that mis-ordering has happened, and an opportunity to…

This tends not to be done in the routers but in the applications.

>
> I agree, however, that there may be a blind spot in the drafts with this regard (generated by the authors' over-familiarity with the subject).  Some mention of in-order delivery might be useful, along the following lines:  A. For many DetNet applications, the transmission interval for any given flow is longer than the delivery time, so the opportunity for out-of-order delivery is strictly limited, and can be engineered out.
I agree that it is limited, I do not currently hold the degree of 
certainty that it can be engineered out expressed in this statement.

> B. Given point A, the use of dual fixed paths and rep/elim, along with worst-case latency guarantees, provides useful limits on out-of-order delivery, even when the paths are much extended by DetNet.  Whether this belongs in architecture, dp-sol, or some other draft is a good question.
... but text on this subject clearly needs to go somewhere.

- Stewart