Subject: Re: [ippm] draft-ioametal-ippm-6man-ioam-ipv6-deployment-00 feedback (Re: v6 option types for IOAM data fields)

From: Mikael Abrahamsson <[email-masked]>
Date: Mon, 1 Apr 2019 13:16:26 +0200 (CEST)

On Thu, 28 Mar 2019, Frank Brockners (fbrockne) wrote:

>
> FYI - A new draft including Mark's comments below just got posted:
> https://tools.ietf.org/html/draft-ioametal-ippm-6man-ioam-ipv6-deployment-01

I watched the tech deep dive video recording in conjunction with this. 
This is a really hard problem to solve with all those factors to take into 
account. Some random thoughts:

There seems to be overlap of this problem space with SRv6, these seem to 
have almost identical properties. They insert something, the forwarding 
devices need to interact with it and it needs to be stripped before it 
leaves the domain.

Though, in the SRv6 space the forwarding information is in the header, 
whereas for the IOAM space the forwarding device should actually ignore 
whatever is in the IOAM header as the desire is for it to treat packets 
identically regardless if they have the header or not. Considering how 
routers forward packets based on 5-tuple (or even deeper), what's the 
thinking been on this so far?

The MTU issue should be the least problematic, recommendations on using a 
considerable higher MTU on the internal links compared to the external 
links is identical to what has been used for encapsulation networks for a 
long time. We use 9180 IP MTU on the internal links and external links are 
9000 MTU. This leaves room for several layers of encap without affecting 
the customer visible MTU.

I like the fact that the IOAM information is attached to the real packet 
being forwarded as opposed to just using test traffic.

I think the C1-C6 requirements make a lot of sense, but following them is 
hard. I have myself run networks that ran IPv4 and IPv6 Internet packets 
native (no encap) internally, and only encapsulated VPN overlay packets. 
It'd of course be nice to support IOAM also for those deployment 
scenarios, but then it becomes harder to assure no leakage (C5) and 
identifying the leaker. Otoh when doing MPLS if you leak labeled packets 
they are probably dropped at the other end (do we have research on this?). 
So leaking IOAM enabled packets might be less of a disaster compared to 
other encaps?

Doing IOAM only based on some other forwarding scheme (MPLS, SRv6 etc) and 
embedding meaning into whatever forwarding information is used for that 
might help (IOAM information lives within that forwarding plane), but also 
makes IOAM less deployable for other networks that just run native.

Reading https://tools.ietf.org/html/draft-ietf-ippm-ioam-data-05 and 
considering the C5 requirement, how do I as an end receiver of a packet 
with an IOAM header in it identify even what operator to contact about 
these leaked packets? It looks like the namespace/identifier is 
operator-internal, but how do I identify what operator it is?

-- 
Mikael Abrahamsson    email: [email-masked]

---

From: "Frank Brockners (fbrockne)" <[email-masked]>
Date: Mon, 1 Apr 2019 16:25:15 +0000

Hi Mikael,

Thanks - please find some notes inline ("... FB").

-----Original Message-----
From: Mikael Abrahamsson <[email-masked]> 
Sent: Montag, 1. April 2019 13:16
To: Frank Brockners (fbrockne) <[email-masked]>
Cc: Mark Smith <[email-masked]>; C. M. Heard <[email-masked]>; 6man WG <[email-masked]>; [email-masked]
Subject: RE: draft-ioametal-ippm-6man-ioam-ipv6-deployment-00 feedback (Re: v6 option types for IOAM data fields)

I watched the tech deep dive video recording in conjunction with this. 
This is a really hard problem to solve with all those factors to take into account. Some random thoughts:

There seems to be overlap of this problem space with SRv6, these seem to have almost identical properties. They insert something, the forwarding devices need to interact with it and it needs to be stripped before it leaves the domain.

Though, in the SRv6 space the forwarding information is in the header, whereas for the IOAM space the forwarding device should actually ignore whatever is in the IOAM header as the desire is for it to treat packets identically regardless if they have the header or not. Considering how routers forward packets based on 5-tuple (or even deeper), what's the thinking been on this so far?

...FB: There is obviously no easy answer. Couple of thoughts:
* IOAM is considered a "domain specific" feature (see draft-ietf-ippm-ioam-data-05, section 3). Routers in the IOAM domain should be IOAM capable.  And IMHO,  we should add a statement to draft-ioametal-ippm-6man-ioam-ipv6-deployment that an implementation of IOAM MUST ensure "C1".
* That said, there can be situations, where C1 cannot be achieved, e.g. consider a network which does ECMP scheduling based on packet length. 
* What one could consider - and which is one suggested deployment model - is that by default IOAM data fields are added to _all_ customer packets. The decapsulating node would decide whether the IOAM information contained in a packet would be used (and exported) or not. That way one would not need to deal with the situation that some traffic carries IOAM while other does not - and might thus be treated differently. 

The MTU issue should be the least problematic, recommendations on using a considerable higher MTU on the internal links compared to the external links is identical to what has been used for encapsulation networks for a long time. We use 9180 IP MTU on the internal links and external links are
9000 MTU. This leaves room for several layers of encap without affecting the customer visible MTU.

I like the fact that the IOAM information is attached to the real packet being forwarded as opposed to just using test traffic.

I think the C1-C6 requirements make a lot of sense, but following them is hard. I have myself run networks that ran IPv4 and IPv6 Internet packets native (no encap) internally, and only encapsulated VPN overlay packets. 
It'd of course be nice to support IOAM also for those deployment scenarios, but then it becomes harder to assure no leakage (C5) and identifying the leaker. Otoh when doing MPLS if you leak labeled packets they are probably dropped at the other end (do we have research on this?). 
So leaking IOAM enabled packets might be less of a disaster compared to other encaps?

...FB: The comparison to MPLS is interesting. How often do MPLS packets leak and cause harm? For IOAM, draft-ioametal-ippm-6man-ioam-ipv6-options-02 proposes a deployment model similar to what we do for MPLS: Unless an interface is explicitly configured for IOAM, packets with IOAM data fields MUST be dropped. Hence leakage would only occur, if we have a clearly misbehaving router which violates this rule. Similar to you, I'd also greatly appreciate any pointers to research on how common MPLS leaked packets are.

Doing IOAM only based on some other forwarding scheme (MPLS, SRv6 etc) and embedding meaning into whatever forwarding information is used for that might help (IOAM information lives within that forwarding plane), but also makes IOAM less deployable for other networks that just run native.

...FB: Yes - this is a trade-off. In some deployments, it might not be feasible to deploy IOAM in the underlay (e.g. v6) - but you still want to use it to measure the overlay (e.g. with NSH) - hence we're defining encapsulations for several "tunnel" protocols. 

Reading https://tools.ietf.org/html/draft-ietf-ippm-ioam-data-05 and considering the C5 requirement, how do I as an end receiver of a packet with an IOAM header in it identify even what operator to contact about these leaked packets? It looks like the namespace/identifier is operator-internal, but how do I identify what operator it is?

...FB: One idea that Shwetha came up with to identify the source AS of a leaked packet, would be to add a new new IOAM E2E option - as proposed in section 5.1.1 bullet 2 of draft-ioametal-ippm-6man-ioam-ipv6-deployment-01.

Cheers, Frank

---

From: Mikael Abrahamsson <[email-masked]>
Date: Tue, 2 Apr 2019 08:05:43 +0200 (CEST)

On Mon, 1 Apr 2019, Frank Brockners (fbrockne) wrote:

> * IOAM is considered a "domain specific" feature (see 
> draft-ietf-ippm-ioam-data-05, section 3). Routers in the IOAM domain 
> should be IOAM capable.  And IMHO,  we should add a statement to 
> draft-ioametal-ippm-6man-ioam-ipv6-deployment that an implementation of 
> IOAM MUST ensure "C1".
>
> * That said, there can be situations, where C1 cannot be achieved, e.g. 
> consider a network which does ECMP scheduling based on packet length.
>
> * What one could consider - and which is one suggested deployment model 
> - is that by default IOAM data fields are added to _all_ customer 
> packets. The decapsulating node would decide whether the IOAM 
> information contained in a packet would be used (and exported) or not. 
> That way one would not need to deal with the situation that some traffic 
> carries IOAM while other does not - and might thus be treated 
> differently.

Yes, I think this is the only way. Is there a risk that intermediate 
routers would not be IOAM capable? I think the C1 requirement is really 
really hard to fulfil and I'm also afraid that adding the IOAM header will 
actually make ECMP stop working on some platforms (because they would not 
have the capability to look deep enough into the packet to find L4 
information, so it'll go back to 2 tuple ECMP instead of 5 tuple.

But this question can only be answered by people with deep NPU 
knowledge...

> ...FB: The comparison to MPLS is interesting. How often do MPLS packets 
> leak and cause harm? For IOAM, 
> draft-ioametal-ippm-6man-ioam-ipv6-options-02 proposes a deployment 
> model similar to what we do for MPLS: Unless an interface is explicitly 
> configured for IOAM, packets with IOAM data fields MUST be dropped. 
> Hence leakage would only occur, if we have a clearly misbehaving router 
> which violates this rule. Similar to you, I'd also greatly appreciate 
> any pointers to research on how common MPLS leaked packets are.

When it comes to "cause harm" I imagine there are (at least) two ways to 
cause harm, one is privacy/secrecy/security loss and the other one is 
actual operational loss.

I know of bugs where labeled packets went the wrong way and caused them to 
be lost, I've also seen bugs where bugs caused traffic to "hop" between 
VRFs in MPLS VPN (or to "global" VRF). I don't have numbers on this 
though.

Depending on the deployment scenario, it might make sense to make IOAM 
packets not valid for non-IOAM aware devices (basically encap entire 
packet/payload), but that might be a problem for intermediate non-IOAM 
nodes then. This would affect the ECMP requirement. I can see cases where 
one would operationally turn on IOAM for some customers traffic and then 
see the problem go away because now ECMP changed.

> ...FB: One idea that Shwetha came up with to identify the source AS of a 
> leaked packet, would be to add a new new IOAM E2E option - as proposed 
> in section 5.1.1 bullet 2 of 
> draft-ioametal-ippm-6man-ioam-ipv6-deployment-01.

Yes, that'd solve that problem.

How much has the silicon people been involved so far in the design of the 
headers? What is the current thinking of amount of data going into the 
IOAM header? Considering things like trace options etc it seems to me the 
header can grow quite large? To satisfy the ECMP requirement what about 
putting the IOAM information as a trailer behind the regular payload? Or 
is there a problem for the hw to manipulate information that far into the 
packet (I also imagine this will considerably lower the forwarding 
performance of IOAM packets on IOAM aware platforms).

---

From: Tom Herbert <[email-masked]>
Date: Tue, 2 Apr 2019 08:26:21 +0200

> But this question can only be answered by people with deep NPU
Mikael,

This is solved with proper use of the IPv6 flow label for ECMP instead
of devices continuing to try to find 5 tuples buried deep within
packets. DPI is only going to become harder as more headers and
functionality are added (e.g. segment routing header will be another
problem, deep headers will exceed parsing buffers, and it's impossible
to find the 5 tuple for a fragment and still be stateless). Since IOAM
is used in a limited domain, the requirement should be that all
routers in the domain support ECMP routing using flow label (note that
is a less stringent requirement than all routers having to support
IOAM).

Tom

> --------------------------------------------------------------------
> IETF IPv6 working group mailing list
> [email-masked]
> Administrative Requests: https://www.ietf.org/mailman/listinfo/ipv6
> --------------------------------------------------------------------

---

From: Mikael Abrahamsson <[email-masked]>
Date: Tue, 2 Apr 2019 09:33:12 +0200 (CEST)

On Tue, 2 Apr 2019, Tom Herbert wrote:

> is used in a limited domain, the requirement should be that all routers 
> in the domain support ECMP routing using flow label (note that is a less 
> stringent requirement than all routers having to support IOAM).

>From what I have seen so far, most devices if they support flow label for 
ECMP, can/will use the flow label in *addition* to 5-tuple.

For the requirement of IOAM to not have separate ECMP characteristics then 
the entire IOAM domain will have to be configured to use 2-tuple + flow 
label?

Do we have data on how much end systems use flow labels nowadays? If this 
is low, do we have data from OS vendors that they're considering 
increasing the use of flow labels?

---

From: "Frank Brockners (fbrockne)" <[email-masked]>
Date: Tue, 2 Apr 2019 09:39:49 +0000

-----Original Message-----
From: Mikael Abrahamsson <[email-masked]> 
Sent: Dienstag, 2. April 2019 08:06
To: Frank Brockners (fbrockne) <[email-masked]>
Cc: Mark Smith <[email-masked]>; C. M. Heard <[email-masked]>; 6man WG <[email-masked]>; [email-masked]
Subject: RE: draft-ioametal-ippm-6man-ioam-ipv6-deployment-00 feedback (Re: v6 option types for IOAM data fields)

On Mon, 1 Apr 2019, Frank Brockners (fbrockne) wrote:

> draft-ioametal-ippm-6man-ioam-ipv6-deployment that an implementation 
> of IOAM MUST ensure "C1".
>
> That way one would not need to deal with the situation that some 
> traffic carries IOAM while other does not - and might thus be treated 
> differently.

Yes, I think this is the only way. Is there a risk that intermediate routers would not be IOAM capable? I think the C1 requirement is really really hard to fulfil and I'm also afraid that adding the IOAM header will actually make ECMP stop working on some platforms (because they would not have the capability to look deep enough into the packet to find L4 information, so it'll go back to 2 tuple ECMP instead of 5 tuple.

But this question can only be answered by people with deep NPU knowledge...

...FB2: Given that IOAM is a domain specific feature - I expect that folks initially start to use it in domains (like e.g. a DC) where all boxes are IOAM capable and can trace the packet appropriately - or per Tom's note, can be configured so that one would do ECMP based on addresses and flow-label.  There are chips with pretty deep parsing capability (up to a few hundred bytes). 

> ...FB: The comparison to MPLS is interesting. How often do MPLS 
> packets leak and cause harm? For IOAM,
> model similar to what we do for MPLS: Unless an interface is 
> explicitly configured for IOAM, packets with IOAM data fields MUST be dropped.
> Hence leakage would only occur, if we have a clearly misbehaving 
> router which violates this rule. Similar to you, I'd also greatly 
> appreciate any pointers to research on how common MPLS leaked packets are.

When it comes to "cause harm" I imagine there are (at least) two ways to cause harm, one is privacy/secrecy/security loss and the other one is actual operational loss.

I know of bugs where labeled packets went the wrong way and caused them to be lost, I've also seen bugs where bugs caused traffic to "hop" between VRFs in MPLS VPN (or to "global" VRF). I don't have numbers on this though.

Depending on the deployment scenario, it might make sense to make IOAM packets not valid for non-IOAM aware devices (basically encap entire packet/payload), but that might be a problem for intermediate non-IOAM nodes then. This would affect the ECMP requirement. I can see cases where one would operationally turn on IOAM for some customers traffic and then see the problem go away because now ECMP changed.

> ...FB: One idea that Shwetha came up with to identify the source AS of 
> a leaked packet, would be to add a new new IOAM E2E option - as 
> proposed in section 5.1.1 bullet 2 of 

How much has the silicon people been involved so far in the design of the headers? What is the current thinking of amount of data going into the IOAM header? Considering things like trace options etc it seems to me the header can grow quite large? To satisfy the ECMP requirement what about putting the IOAM information as a trailer behind the regular payload? Or is there a problem for the hw to manipulate information that far into the packet (I also imagine this will considerably lower the forwarding performance of IOAM packets on IOAM aware platforms).

...FB2: There are quite a few folks with hardware background that co-author the IOAM drafts. Parsing capability differs between chips, as does the ability to deal with new/flexible headers and the ability to modify data in the extension headers. So the unfortunate answer to that question is "it depends" (what information do you gather, how large is your domain, what are the capabilities of your hardware/software forwarder, etc.), but I do expect that for specific deployment domains (e.g. DC, Enterprise) - but also for overlay networks / VPNs - we'll see an increasing amount of chips that are well suited for processing large extension header.

Cheers, Frank

---

From: Tom Herbert <[email-masked]>
Date: Tue, 2 Apr 2019 16:45:26 +0200

Mikael,

They can do that if they want, but it's less reliable then just using
the flow label and three tuple. There is no gaurantee that the five
tuple can be consistently determined for all packets of a flow for a
number of reasons. Even with the just the IOAM in an EH, there are
going to be devices that don't have the logic to skip over an EH to
find the transport layer, so that motivates that the idea that every
packet needs to have the IOAM attached whether it's needed or not
(hence it is an overhead tax).

> the entire IOAM domain will have to be configured to use 2-tuple + flow
It is a SHOULD. IMO, it's a less invasive requirement than requiring
all routers in the domain to support IOAM or that IOAM must be set in
every packet.

> increasing the use of flow labels?

All major OSes (Windows, iOS, FreeBSD, Linux) now set the label by
default per RFC6437 and the flow label is usable for ECMP as described
in RFC6438. Linux has been automatically setting flow labels since
2014. So they are definitely being set on packets in the Internet. I'm
not sure if anyone has yet done measurements on how pervasive they
are, but it would be easy enough for a major application or service
provider to count them.

I should also point out that even if the flow label is not set, ECMP
still works and gives consistent results where a "flow" is all packets
sent from one IP address to another. The granularity of doing ECMP per
transport flow is really only needed when there are a lot of flows
over some IP address pair like in a tunnel for instance. There is a
lot of incentive to use flow labels with tunnels to get ECMP. For a
tunnel packet, the flow label is set as a hash for flow of an
encapsulated packet and ECMP is effective just by using the three
tuple of fields in the IP header. For a device trying to extract the
5-tuple they would need to parse over encapsulation shim layers to
find the transport layer. Again, finding the 5-tuple is considerable
work and there's no guarantee that parsing a particular encapsulation
protocol (there are a lot of them) is supported by all devices or that
inner five tuple is even observable (e.g. the tunnel could be
encrypted).

Tom

---

From: "Pengshuping (Peng Shuping)" <[email-masked]>
Date: Wed, 3 Apr 2019 13:46:57 +0000

Hi, 

The parsing depth of the early chips is 96B/128B. For the new chips the parsing depth has been increased but still limited. So Mikael's concern makes sense especially in the tracing mode that the IOAM data fields are going to increase significantly along the path, which will even push the Routing Header out of the parsing depth of the chip. So it would make sense to split the IOAM data part from the IOAM header/instruction part, and place the IOAM data after the RH or even after the L4 header in order to be able to read the L4 information to enable ECMP, as stated in the draft https://tools.ietf.org/html/draft-li-6man-ipv6-sfc-ifit-00.

BR,
Shuping 

-----邮件原件-----
发件人: ippm [mailto:[email-masked]] 代表 Frank Brockners (fbrockne)
发送时间: 2019年4月2日 17:40
收件人: Mikael Abrahamsson <[email-masked]>
抄送: C. M. Heard <[email-masked]>; 6man WG <[email-masked]>; Mark Smith <[email-masked]>; [email-masked]
主题: Re: [ippm] draft-ioametal-ippm-6man-ioam-ipv6-deployment-00 feedback (Re: v6 option types for IOAM data fields)

-----Original Message-----
From: Mikael Abrahamsson <[email-masked]>
Sent: Dienstag, 2. April 2019 08:06
To: Frank Brockners (fbrockne) <[email-masked]>
Cc: Mark Smith <[email-masked]>; C. M. Heard <[email-masked]>; 6man WG <[email-masked]>; [email-masked]
Subject: RE: draft-ioametal-ippm-6man-ioam-ipv6-deployment-00 feedback (Re: v6 option types for IOAM data fields)

....FB2: Given that IOAM is a domain specific feature - I expect that folks initially start to use it in domains (like e.g. a DC) where all boxes are IOAM capable and can trace the packet appropriately - or per Tom's note, can be configured so that one would do ECMP based on addresses and flow-label.  There are chips with pretty deep parsing capability (up to a few hundred bytes). 

....FB2: There are quite a few folks with hardware background that co-author the IOAM drafts. Parsing capability differs between chips, as does the ability to deal with new/flexible headers and the ability to modify data in the extension headers. So the unfortunate answer to that question is "it depends" (what information do you gather, how large is your domain, what are the capabilities of your hardware/software forwarder, etc.), but I do expect that for specific deployment domains (e.g. DC, Enterprise) - but also for overlay networks / VPNs - we'll see an increasing amount of chips that are well suited for processing large extension header.

Cheers, Frank

_______________________________________________
ippm mailing list
[email-masked]
https://www.ietf.org/mailman/listinfo/ippm

---

From: Barak Gafni <[email-masked]>
Date: Thu, 4 Apr 2019 19:17:46 +0000

Hi Shuping,
I think that the question is whether we design for the lowest capable solutions. I assume someone can find solutions with even lower amount of parsing. The low end solutions for these capabilities may need to either adopt with new designs or use solutions such as what is suggested in this thread by Tom Herbert.
Personally, I don't think we should go this way, but to design appropriate solutions, with appropriate layers in the headers. For example, prevent dependencies between "remote" headers.

Thanks,
Barak

-----Original Message-----
From: ippm <[email-masked]> On Behalf Of Pengshuping (Peng Shuping)
Sent: Wednesday, April 3, 2019 6:47 AM
To: Frank Brockners (fbrockne) <[email-masked]>; Mikael Abrahamsson <[email-masked]>
Cc: C. M. Heard <[email-masked]>; 6man WG <[email-masked]>; Mark Smith <[email-masked]>; [email-masked]
Subject: [ippm] 答复: draft-ioametal-ippm-6man-ioam-ipv6-deployment-00 feedback (Re: v6 option types for IOAM data fields)

Hi, 

The parsing depth of the early chips is 96B/128B. For the new chips the parsing depth has been increased but still limited. So Mikael's concern makes sense especially in the tracing mode that the IOAM data fields are going to increase significantly along the path, which will even push the Routing Header out of the parsing depth of the chip. So it would make sense to split the IOAM data part from the IOAM header/instruction part, and place the IOAM data after the RH or even after the L4 header in order to be able to read the L4 information to enable ECMP, as stated in the draft https://eur03.safelinks.protection.outlook.com/?url=https%3A%2F%2Ftools.ietf.org%2Fhtml%2Fdraft-li-6man-ipv6-sfc-ifit-00&amp;data=02%7C01%7Cgbarak%40mellanox.com%7C612ba6dc117440e68c4608d6b83ae174%7Ca652971c7d2e4d9ba6a4d149256f461b%7C0%7C0%7C636898960379485833&amp;sdata=Dob4zN%2F1jSGxLTlqGma29JJNvqJDBt4VzqHsN4OpxM4%3D&amp;reserved=0.

BR,
Shuping 

-----邮件原件-----
发件人: ippm [mailto:[email-masked]] 代表 Frank Brockners (fbrockne)
发送时间: 2019年4月2日 17:40
收件人: Mikael Abrahamsson <[email-masked]>
抄送: C. M. Heard <[email-masked]>; 6man WG <[email-masked]>; Mark Smith <[email-masked]>; [email-masked]
主题: Re: [ippm] draft-ioametal-ippm-6man-ioam-ipv6-deployment-00 feedback (Re: v6 option types for IOAM data fields)

-----Original Message-----
From: Mikael Abrahamsson <[email-masked]>
Sent: Dienstag, 2. April 2019 08:06
To: Frank Brockners (fbrockne) <[email-masked]>
Cc: Mark Smith <[email-masked]>; C. M. Heard <[email-masked]>; 6man WG <[email-masked]>; [email-masked]
Subject: RE: draft-ioametal-ippm-6man-ioam-ipv6-deployment-00 feedback (Re: v6 option types for IOAM data fields)

Cheers, Frank

_______________________________________________
ippm mailing list
[email-masked]
https://eur03.safelinks.protection.outlook.com/?url=https%3A%2F%2Fwww.ietf.org%2Fmailman%2Flistinfo%2Fippm&amp;data=02%7C01%7Cgbarak%40mellanox.com%7C612ba6dc117440e68c4608d6b83ae174%7Ca652971c7d2e4d9ba6a4d149256f461b%7C0%7C0%7C636898960379485833&amp;sdata=VjZ1PiwifYGCf159je0yef3BJFz6AQNNAtkgmAMDFKM%3D&amp;reserved=0
_______________________________________________
ippm mailing list
[email-masked]
https://eur03.safelinks.protection.outlook.com/?url=https%3A%2F%2Fwww.ietf.org%2Fmailman%2Flistinfo%2Fippm&amp;data=02%7C01%7Cgbarak%40mellanox.com%7C612ba6dc117440e68c4608d6b83ae174%7Ca652971c7d2e4d9ba6a4d149256f461b%7C0%7C0%7C636898960379485833&amp;sdata=VjZ1PiwifYGCf159je0yef3BJFz6AQNNAtkgmAMDFKM%3D&amp;reserved=0

---

From: "Pengshuping (Peng Shuping)" <[email-masked]>
Date: Fri, 5 Apr 2019 16:15:57 +0000

Hi Barak,

We are certainly not designing for the lowest capable solutions but exploring optimal solutions which could help to keep the forwarding performance while inserting the iOAM data with variable length.

In terms of the forwarding efficiency, we would all agree that a short and fixed header would be better than a variable header.

Best regards
Shuping

发件人： Barak Gafni<[email-masked]<mailto:[email-masked]>>
收件人： Pengshuping (Peng Shuping)<[email-masked]<mailto:[email-masked]>>;Frank Brockners (fbrockne)<[email-masked]<mailto:[email-masked]>>;Mikael Abrahamsson<[email-masked]<mailto:[email-masked]>>
抄送： C. M. Heard<[email-masked]<mailto:[email-masked]>>;6man WG<[email-masked]<mailto:[email-masked]>>;Mark Smith<[email-masked]<mailto:[email-masked]>>;ippm<[email-masked]<mailto:[email-masked]>>
主题： RE: draft-ioametal-ippm-6man-ioam-ipv6-deployment-00 feedback (Re: v6 option types for IOAM data fields)
时间： 2019-04-05 03:18:26

Hi Shuping,

Thanks,
Barak

-----Original Message-----
From: ippm <[email-masked]> On Behalf Of Pengshuping (Peng Shuping)
Sent: Wednesday, April 3, 2019 6:47 AM
To: Frank Brockners (fbrockne) <[email-masked]>; Mikael Abrahamsson <[email-masked]>
Cc: C. M. Heard <[email-masked]>; 6man WG <[email-masked]>; Mark Smith <[email-masked]>; [email-masked]
Subject: [ippm] 答复: draft-ioametal-ippm-6man-ioam-ipv6-deployment-00 feedback (Re: v6 option types for IOAM data fields)

Hi,

BR,
Shuping

-----邮件原件-----
发件人: ippm [mailto:[email-masked]] 代表 Frank Brockners (fbrockne)
发送时间: 2019年4月2日 17:40
收件人: Mikael Abrahamsson <[email-masked]>
抄送: C. M. Heard <[email-masked]>; 6man WG <[email-masked]>; Mark Smith <[email-masked]>; [email-masked]
主题: Re: [ippm] draft-ioametal-ippm-6man-ioam-ipv6-deployment-00 feedback (Re: v6 option types for IOAM data fields)

-----Original Message-----
From: Mikael Abrahamsson <[email-masked]>
Sent: Dienstag, 2. April 2019 08:06
To: Frank Brockners (fbrockne) <[email-masked]>
Cc: Mark Smith <[email-masked]>; C. M. Heard <[email-masked]>; 6man WG <[email-masked]>; [email-masked]
Subject: RE: draft-ioametal-ippm-6man-ioam-ipv6-deployment-00 feedback (Re: v6 option types for IOAM data fields)

Cheers, Frank

---

From: Tom Herbert <[email-masked]>
Date: Fri, 5 Apr 2019 10:31:19 -0700

> We are certainly not designing for the lowest capable solutions but exploring optimal solutions which could help to keep the forwarding performance while inserting the iOAM data with variable lengt…

Shuping,

That's exactly the argument for using the flow label in ECMP.
Efficeint packet forwarding can be done solely by inspected the forty
byte fixed length header of the packet regardless of the contents of
the rest of the packet. That is an optimal solution. Anything else for
ECMP like doing DPI or hacking up IOAM to try do pull transport layers
into the parsing buffer (whatever size that may be) are nothing more
than hacks and unnecessarily perpetuate techniques used with IPv4 that
either don't work or are unnecessary in IPv6.

Tom

> 发件人： Barak Gafni<[email-masked]>
> 收件人： Pengshuping (Peng Shuping)<[email-masked]>;Frank Brockners (fbrockne)<[email-masked]>;Mikael Abrahamsson<[email-masked]>
> 抄送： C. M. Heard<[email-masked]>;6man WG<[email-masked]>;Mark Smith<[email-masked]>;ippm<[email-masked]>
> I think that the question is whether we design for the lowest capable solutions. I assume someone can find solutions with even lower amount of parsing.. The low end solutions for these capabilities may need to either adopt with new designs or use solutions such as what is suggested in this thread by Tom Herbert.
> > * That said, there can be situations, where C1 cannot be achieved, e.g..
> .....FB2: Given that IOAM is a domain specific feature - I expect that folks initially start to use it in domains (like e.g. a DC) where all boxes are IOAM capable and can trace the packet appropriately - or per Tom's note, can be configured so that one would do ECMP based on addresses and flow-label.  There are chips with pretty deep parsing capability (up to a few hundred bytes).
>
> .....FB2: There are quite a few folks with hardware background that co-author the IOAM drafts. Parsing capability differs between chips, as does the ability to deal with new/flexible headers and the ability to modify data in the extension headers. So the unfortunate answer to that question is "it depends" (what information do you gather, how large is your domain, what are the capabilities of your hardware/software forwarder, etc.), but I do expect that for specific deployment domains (e.g. DC, Enterprise) - but also for overlay networks / VPNs - we'll see an increasing amount of chips that are well suited for processing large extension header.
>
> Cheers, Frank
>
> --

---

From: Jai Kumar <[email-masked]>
Date: Fri, 05 Apr 2019 10:55:20 -0700

Even if we use flow label for ECMP, lack of accessibility of L4 information (for various services, most simple is ACL) makes the proposal pretty much unusable.

NIC cards are not used pervasively for ACLs and other services, I believe iptables in kernel are good enough for host but that’s not acceptable in networking elements.

-Jai

On 4/5/19, 10:31 AM, "ippm on behalf of Tom Herbert" <[email-masked] on behalf of [email-masked]> wrote:

    Shuping,

    Tom

---

From: Tom Herbert <[email-masked]>
Date: Fri, 5 Apr 2019 11:15:34 -0700

> Even if we use flow label for ECMP, lack of accessibility of L4 information (for various services, most simple is ACL) makes the proposal pretty much unusable.
That is convoluting router and firewall functionality and
requirements. This a discussion about packet forwarding which does
_not_ require ACLs. In a limited domain, for which IOAM is intended,
it is unlikely that they'll ACLs would be used in internal nodes, but
we do need good ECMP routing at all intermediate nodes need to be
consistent rather or not that the IOAM option is present. As side
note, the ability to extract L4 information out of packets is
dwindling anyway thanks to encryption and other techniques trying to
undo protocol ossification. For instance, good luck getting L4
information out of an IPsec tunnel :-).

Tom

---

From: Jai Kumar <[email-masked]>
Date: Fri, 05 Apr 2019 11:25:52 -0700

Tom,

I am speaking based on the actual deployment not based on if all the traffic is/will move to IPSec tunnel.

I have 6 large customer deployments and one someone previously called as "Yahoo" who insist on visibility in L4 even for IOAM packets.

I can quote you on that and lose my business :-)

ECMP argument is weak as well as it works for IPv6 flow label. What about IPv4 TCP/UDP traffic which typically is 80% of the traffic in MSDC.

If someone tells me that proposal will work only for my 20% of traffic then again as I said before, they will lose my business :-)

Regards,
-Jai

On 4/5/19, 11:15 AM, "Tom Herbert" <[email-masked]> wrote:

    Tom

---

From: Tom Herbert <[email-masked]>
Date: Fri, 5 Apr 2019 11:54:09 -0700

> I have 6 large customer deployments and one someone previously called as "Yahoo" who insist on visibility in L4 even for IOAM packets.
And I've worked in datcenters much larger, and my first rule is that I
don't want random router vendors dictating to me what protocols I'm
allowed to use in my datacenter.

> ECMP argument is weak as well as it works for IPv6 flow label. What about IPv4 TCP/UDP traffic which typically is 80% of the traffic in MSDC.

draft-herbert-ipv4-udpencap-eh-01

> If someone tells me that proposal will work only for my 20% of traffic then again as I said before, they will lose my business :-)
Great, but this IETF. Where are the protocol requirements? If you are
saying that there are requirements for "visibility in L4" what are
they? If you are saying that ACLs are now required for packet
forwarding, where is that described? Is it a MUST that teh only
transport protocols we're allowed to use are  UDP or TCP? Are we
allowed to use extension headers at all? I suppose fragmentation is
right out... Are we violating the requirements if we encrypt the
transport headers? If you say that we shouldn't send long headers
because we'll overflow parsing buffers, then what is the limit? Please
be SPECIFC in setting requirements; we are trying to build protocols
that work and are interoperable and we need to get out of this morass
of reverse engineering to discover the least common denominator.

Tom

---

From: Jai Kumar <[email-masked]>
Date: Fri, 05 Apr 2019 12:03:22 -0700

" Great, but this IETF"
Does it mean that drafts have no relevance to deployment and actual adoption.

For all your queries please take a look at IFA draft.
https://datatracker.ietf.org/doc/draft-kumar-ippm-ifa/

It handles IPv6, IPv4 TCP/UDP with altering the packet path (due to inserting options and/or encaping them in GRE header) as well as IPSec AH/ESP/WESP both tunnel and transport and guess what it is successfully deployed. 

-Jai

On 4/5/19, 11:54 AM, "Tom Herbert" <[email-masked]> wrote:

    

    Tom

---

From: Jai Kumar <[email-masked]>
Date: Fri, 05 Apr 2019 12:06:33 -0700

It handles IPv6, IPv4 TCP/UDP *without altering the packet path ...

On 4/5/19, 12:03 PM, "Jai Kumar" <[email-masked]> wrote:

    " Great, but this IETF"

    
    -Jai

        

        
        Tom

---

From: Tom Herbert <[email-masked]>
Date: Fri, 5 Apr 2019 12:27:50 -0700

> https://datatracker.ietf.org/doc/draft-kumar-ippm-ifa/
I've looked at that draft. It's is creating a complete new IP
protocolas well as a new packet format for TCP and UDP that somehow
inserts meta data between the TCP header and payload. I don't
understand why this could possibly be better than just using flow
label for ECMP and teaching devices that need to extract transport
information how to walk over EH, or just using some UDP encpasulation
like in draft-herbert-ipv4-udpencap-eh-01.

Tom

---

From: Jai Kumar <[email-masked]>
Date: Fri, 05 Apr 2019 14:07:22 -0700

Tom,

Its all explained in the requirements section of the draft. IFA header is essentially treated as an EH both in IPv4 and IPv6 (very similar to AH/ESP EH).

Just like in AH/ESP the IP protocol is copied in the AH header and IP protocol field is replaced in the IP header with the AH proto.

There are several advantages to this approach.
- devise know how to parse AH header and can follow very similar approach
- Protocol agnostics, silicon need not support N encaps (just like AH/ESP). Single implementation works for all protocol types.
- No need to creation options and teach devices that options is NOT an exception path processing
- Really worry about ordering in IPv6 options
- Note that all the OAM data need to be deleted as well so position need to be optimal for removal. We can always do tail stamping of the data but this puts considerable constrains on the cell based architecture
- L4 accessibility I have already mentioned

I am happy to discuss with you in person and explain more but bottom line is that this proposal is what came out with MSDC discussions in light of shortcomings from other IETF proposals.

Regards,
-Jai

On 4/5/19, 12:28 PM, "Tom Herbert" <[email-masked]> wrote:

    Tom

---

From: Tom Herbert <[email-masked]>
Date: Fri, 5 Apr 2019 14:34:35 -0700

On Fri, Apr 5, 2019, 2:07 PM Jai Kumar <[email-masked]> wrote:

> Tom,
>
> Its all explained in the requirements section of the draft. IFA header is
> essentially treated as an EH both in IPv4 and IPv6 (very similar to AH/ESP
> EH).
>
> Just like in AH/ESP the IP protocol is copied in the AH header and IP
> protocol field is replaced in the IP header with the AH proto.
>
> - Protocol agnostics, silicon need not support N encaps (just like
> AH/ESP). Single implementation works for all protocol types.
> - Note that all the OAM data need to be deleted as well so position need
> to be optimal for removal. We can always do tail stamping of the data but
> this puts considerable constrains on the cell based architecture
> I am happy to discuss with you in person and explain more but bottom line
> is that this proposal is what came out with MSDC discussions in light of
> shortcomings from other IETF proposals.
>

As they say, you are free to run whatever you want in your proprietary
network, but if you expect to create an interoperable, deployable protocoI
and get a protocol number assignment, you'll need to work in ietf. That is
going to require justifying the protocol even in light of shortcomings in
existing protocoIs.

Tom

>     > It handles IPv6, IPv4 TCP/UDP with altering the packet path (due to
> inserting options and/or encaping them in GRE header) as well as IPSec
> AH/ESP/WESP both tunnel and transport and guess what it is successfully
> deployed.
>     >
>     > -Jai
>     >
>     >     > I am speaking based on the actual deployment not based on if
> all the traffic is/will move to IPSec tunnel.
>     >     >
>     >     > I have 6 large customer deployments and one someone previously
> called as "Yahoo" who insist on visibility in L4 even for IOAM packets.
>     >     >
>     >     > ECMP argument is weak as well as it works for IPv6 flow label.
> What about IPv4 TCP/UDP traffic which typically is 80% of the traffic in
> MSDC.
>     >     >
>     >
>     >     > If someone tells me that proposal will work only for my 20% of
> traffic then again as I said before, they will lose my business :-)
>     >     >
>     >     >     > Even if we use flow label for ECMP, lack of
> accessibility of L4 information (for various services, most simple is ACL)
> makes the proposal pretty much unusable.
>     >     >     >
>     >     >     > NIC cards are not used pervasively for ACLs and other
> services, I believe iptables in kernel are good enough for host but that’s
> not acceptable in networking elements.
>     >     >     >
>     >     >     > -Jai
>     >     >     >
>     >     >     > On 4/5/19, 10:31 AM, "ippm on behalf of Tom Herbert" <
> [email-masked] on behalf of [email-masked]> wrote:
>     >     >     >
>     >     >     >     > We are certainly not designing for the lowest
> capable solutions but exploring optimal solutions which could help to keep
> the forwarding performance while inserting the iOAM data with variable
> length.
>     >     >     >
>     >     >     >     Shuping,
>     >     >     >
>     >     >     >     > In terms of the forwarding efficiency, we would
> all agree that a short and fixed header would be better than a variable
> header.
>     >     >     >     >
>     >     >     >     > Best regards
>     >     >     >     > Shuping
>     >     >     >     >
>     >     >     >     >
>     >     >     >     > 收件人： Pengshuping (Peng Shuping)<
> [email-masked]>;Frank Brockners (fbrockne)<[email-masked]>;Mikael
> Abrahamsson<[email-masked]>
>     >     >     >     > 抄送： C. M. Heard<[email-masked]>;6man WG<
> [email-masked]>;Mark Smith<[email-masked]>;ippm<[email-masked]>
>     >     >     >     > 主题： RE:
> draft-ioametal-ippm-6man-ioam-ipv6-deployment-00 feedback (Re: v6 option
> types for IOAM data fields)
>     >     >     >     > 时间： 2019-04-05 03:18:26
>     >     >     >     >
>     >     >     >     > Hi Shuping,
>     >     >     >     > I think that the question is whether we design for
> the lowest capable solutions. I assume someone can find solutions with even
> lower amount of parsing.. The low end solutions for these capabilities may
> need to either adopt with new designs or use solutions such as what is
> suggested in this thread by Tom Herbert.
>     >     >     >     > Personally, I don't think we should go this way,
> but to design appropriate solutions, with appropriate layers in the
> headers. For example, prevent dependencies between "remote" headers.
>     >     >     >     >
>     >     >     >     > Thanks,
>     >     >     >     > Barak
>     >     >     >     >
>     >     >     >     > Cc: C. M. Heard <[email-masked]>; 6man WG <
> [email-masked]>; Mark Smith <[email-masked]>; [email-masked]
>     >     >     >     > Subject: [ippm] 答复:
> draft-ioametal-ippm-6man-ioam-ipv6-deployment-00 feedback (Re: v6 option
> types for IOAM data fields)
>     >     >     >     >
>     >     >     >     > Hi,
>     >     >     >     >
>     >     >     >     > The parsing depth of the early chips is 96B/128B.
> For the new chips the parsing depth has been increased but still limited.
> So Mikael's concern makes sense especially in the tracing mode that the
> IOAM data fields are going to increase significantly along the path, which
> will even push the Routing Header out of the parsing depth of the chip. So
> it would make sense to split the IOAM data part from the IOAM
> header/instruction part, and place the IOAM data after the RH or even after
> the L4 header in order to be able to read the L4 information to enable
> ECMP, as stated in the draft
> https://eur03.safelinks.protection.outlook.com/?url=https%3A%2F%2Ftools.ietf.org%2Fhtml%2Fdraft-li-6man-ipv6-sfc-ifit-00&amp;data=02%7C01%7Cgbarak%40mellanox.com%7C612ba6dc117440e68c4608d6b83ae174%7Ca652971c7d2e4d9ba6a4d149256f461b%7C0%7C0%7C636898960379485833&amp;sdata=Dob4zN%2F1jSGxLTlqGma29JJNvqJDBt4VzqHsN4OpxM4%3D&amp;reserved=0
> .
>     >     >     >     >
>     >     >     >     > BR,
>     >     >     >     > Shuping
>     >     >     >     >
>     >     >     >     >
>     >     >     >     > -----邮件原件-----
>     >     >     >     > 抄送: C. M. Heard <[email-masked]>; 6man WG <
> [email-masked]>; Mark Smith <[email-masked]>; [email-masked]
>     >     >     >     > 主题: Re: [ippm]
> draft-ioametal-ippm-6man-ioam-ipv6-deployment-00 feedback (Re: v6 option
> types for IOAM data fields)
>     >     >     >     >
>     >     >     >     >
>     >     >     >     >
>     >     >     >     > To: Frank Brockners (fbrockne) <[email-masked]
> >
>     >     >     >     > Cc: Mark Smith <[email-masked]>; C. M.
> Heard <[email-masked]>; 6man WG <[email-masked]>; [email-masked]
>     >     >     >     > Subject: RE:
> draft-ioametal-ippm-6man-ioam-ipv6-deployment-00 feedback (Re: v6 option
> types for IOAM data fields)
>     >     >     >     >
>     >     >     >     > > * What one could consider - and which is one
> suggested deployment
>     >     >     >     > > model
>     >     >     >     > Yes, I think this is the only way. Is there a risk
> that intermediate routers would not be IOAM capable? I think the C1
> requirement is really really hard to fulfil and I'm also afraid that adding
> the IOAM header will actually make ECMP stop working on some platforms
> (because they would not have the capability to look deep enough into the
> packet to find L4 information, so it'll go back to 2 tuple ECMP instead of
> 5 tuple.
>     >     >     >     >
>     >     >     >     > .....FB2: Given that IOAM is a domain specific
> feature - I expect that folks initially start to use it in domains (like
> e.g. a DC) where all boxes are IOAM capable and can trace the packet
> appropriately - or per Tom's note, can be configured so that one would do
> ECMP based on addresses and flow-label.  There are chips with pretty deep
> parsing capability (up to a few hundred bytes).
>     >     >     >     >
>     >     >     >     > When it comes to "cause harm" I imagine there are
> (at least) two ways to cause harm, one is privacy/secrecy/security loss and
> the other one is actual operational loss.
>     >     >     >     >
>     >     >     >     > I know of bugs where labeled packets went the
> wrong way and caused them to be lost, I've also seen bugs where bugs caused
> traffic to "hop" between VRFs in MPLS VPN (or to "global" VRF). I don't
> have numbers on this though.
>     >     >     >     >
>     >     >     >     > Depending on the deployment scenario, it might
> make sense to make IOAM packets not valid for non-IOAM aware devices
> (basically encap entire packet/payload), but that might be a problem for
> intermediate non-IOAM nodes then. This would affect the ECMP requirement. I
> can see cases where one would operationally turn on IOAM for some customers
> traffic and then see the problem go away because now ECMP changed.
>     >     >     >     >
>     >     >     >     > How much has the silicon people been involved so
> far in the design of the headers? What is the current thinking of amount of
> data going into the IOAM header? Considering things like trace options etc
> it seems to me the header can grow quite large? To satisfy the ECMP
> requirement what about putting the IOAM information as a trailer behind the
> regular payload? Or is there a problem for the hw to manipulate information
> that far into the packet (I also imagine this will considerably lower the
> forwarding performance of IOAM packets on IOAM aware platforms).
>     >     >     >     >
>     >     >     >     > .....FB2: There are quite a few folks with
> hardware background that co-author the IOAM drafts. Parsing capability
> differs between chips, as does the ability to deal with new/flexible
> headers and the ability to modify data in the extension headers. So the
> unfortunate answer to that question is "it depends" (what information do
> you gather, how large is your domain, what are the capabilities of your
> hardware/software forwarder, etc.), but I do expect that for specific
> deployment domains (e.g. DC, Enterprise) - but also for overlay networks /
> VPNs - we'll see an increasing amount of chips that are well suited for
> processing large extension header.
>     >     >     >     >
>     >     >     >     > Cheers, Frank
>     >     >     >     >
>     >     >     >     > --
>     >     >     >     > Administrative Requests:
> https://www.ietf.org/mailman/listinfo/ipv6
>     >     >     >     >

---

From: Mark Smith <[email-masked]>
Date: Sat, 6 Apr 2019 08:42:45 +1100

On Sat, 6 Apr 2019 at 08:35, Tom Herbert <[email-masked]> wrote:
> As they say, you are free to run whatever you want in your proprietary network, but if you expect to create an interoperable, deployable protocoI and get a protocol number assignment, you'll need to work in ietf. That is going to require justifying the protocol even in light of shortcomings in existing protocoIs.
>

And if it doesn't comply with the IPv6 protocol specification, you
can't call it IPv6, even if the packet structure is the same. "closed
domains" aren't a free for all for packet handing behaviour if you're
also claiming protocol compliance.

>>     >     >     > Even if we use flow label for ECMP, lack of accessibility of L4 information (for various services, most simple is ACL) makes the proposal pretty much unusable..
>>     >     >     >

---

From: Jai Kumar <[email-masked]>
Date: Fri, 05 Apr 2019 15:09:18 -0700

“That is going to require justifying the protocol even in light of shortcomings in existing protocoIs.”

Agreed !!! True for any proposal.

-Jai

From: Tom Herbert <[email-masked]>
Date: Friday, April 5, 2019 at 2:34 PM
To: Jai Kumar <[email-masked]>
Cc: "Pengshuping (Peng Shuping)" <[email-masked]>, 6man <[email-masked]>, ippm <[email-masked]>, "C. M. Heard" <[email-masked]>, Mikael Abrahamsson <[email-masked]>
Subject: Re: [ippm] draft-ioametal-ippm-6man-ioam-ipv6-deployment-00 feedback (Re: v6 option types for IOAM data fields)

Tom,

Tom

Regards,
-Jai

    Tom

---

From: Tom Herbert <[email-masked]>
Date: Fri, 5 Apr 2019 15:34:28 -0700

Jai,

You might want to consider how IFA works in the presence of segment
routing. The second someone puts an SRH in the packet the IFA headers get
pushed down and the transport headers end up being deep in the packet
anyway so that all the effort of IFA is nullified. If the requirement is to
ensure that the transport layer information appear as early as possible in
the packet, one could define a Hop-by-Hop option to carry some sort of
pseudo header for the transport layer. This also would work in the case
that the actual transport layer is obfuscated.

Tom

On Fri, Apr 5, 2019, 3:09 PM Jai Kumar <[email-masked]> wrote:

> *From: *Tom Herbert <[email-masked]>
> *Date: *Friday, April 5, 2019 at 2:34 PM
> *To: *Jai Kumar <[email-masked]>
> *Cc: *"Pengshuping (Peng Shuping)" <[email-masked]>, 6man <
> [email-masked]>, ippm <[email-masked]>, "C. M. Heard" <[email-masked]>,
> Mikael Abrahamsson <[email-masked]>
> *Subject: *Re: [ippm] draft-ioametal-ippm-6man-ioam-ipv6-deployment-00
> feedback (Re: v6 option types for IOAM data fields)
>
>
>
>
>

---

From: Jai Kumar <[email-masked]>
Date: Fri, 05 Apr 2019 16:40:13 -0700

Tom,

You are right. There are two main cases where accessibility of transport header becomes an issue. And that is mainly because of current generation of fixed pipeline cell based architecture parse depth (not so much an issue for high end NPU where next packet header bytes can be DMA’ed in, though it impacts the performance. For eg. Cisco QFP NPU, EZ Chip, Juniper’s MX 3D NPU, or Broadcom’s Jerricho NPU ). This was mentioned earlier by Mikael as well.

Case 1. SRv6

Case 2. IPv6 with EH (mainly for 5G deployments where RH, DH, and IFA Header) 

Typically NPUs provide capability that it will handle 3 or 4 EH or SRH without performance penalty. With the presence of IFA header this means that for eg there can be only 2 or 3 SRH or EH. This is same as in MPLS label stack depth handling (PUSH or POP at PE router). Given the evolution SRV6 and MPLS-SR kind of headers, silicon vendors have taken a notice of it and are evolving the silicon arch to handle more chained headers.

IFA supports fragmentation of metadata stack and also postcard mode so beyond the transport header it’s not an issue.

Just a quick feedback on using IPv6 options, inserting hop by hop options for data path processing pushes the EH further down and impacts the node’s forwarding function (say if it is not able to reach Destination EH) vs if IFA header is the last EH then atmost transit node do not perform metadata collection which can be identified.

-Jai

Date: Friday, April 5, 2019 at 3:34 PM

Jai,

You might want to consider how IFA works in the presence of segment routing. The second someone puts an SRH in the packet the IFA headers get pushed down and the transport headers end up being deep in the packet anyway so that all the effort of IFA is nullified. If the requirement is to ensure that the transport layer information appear as early as possible in the packet, one could define a Hop-by-Hop option to carry some sort of pseudo header for the transport layer. This also would work in the case that the actual transport layer is obfuscated.

Tom

-Jai

 

Tom,

Tom

Regards,
-Jai

    Tom

---

From: Tom Herbert <[email-masked]>
Date: Fri, 5 Apr 2019 17:05:26 -0700

> Just a quick feedback on using IPv6 options, inserting hop by hop options for data path processing pushes the EH further down and impacts the node’s forwarding function (say if it is not able to re…

Maybe it does impact it, maybe it doesn't. It clearly depends on the
device capabilities. For a device with a parsing buffer of 256 bytes
that should be plenty of room for HBH options, SR header, transport
layer, etc.

---

From: Jai Kumar <[email-masked]>
Date: Sat, 06 Apr 2019 09:41:23 -0700

Tom,

Please read the IOAM draft again. There is a variable length metadata stack as well with trace option. Forwarding becomes completely indeterministic.
In short using options forwarding gets impacted for EH. You still want to push for it go by all means but don't expect a silicon vendor or deployment to agree for it.

-Jai

On 4/5/19, 5:05 PM, "Tom Herbert" <[email-masked]> wrote:

---

From: Tom Herbert <[email-masked]>
Date: Sat, 6 Apr 2019 12:55:20 -0700

Jai,

Use the flow label and addresses as input to ECMP forwarding _is_
deterministic. And not just for deterministic for TCP or UDP, but for
_any_ protocol, combination of EH, encryptions, etc. that a packet
carries. See RFC6438.

> In short using options forwarding gets impacted for EH. You still want to push for it go by all means but don't expect a silicon vendor or deployment to agree for it.

I'm not sure what we're trying to get agreement on. The only
expectation we have is that vendors, and everyone else for that
matter, conform to IETF standards. Extension headers have be defined
for over twenty years and are full standard, and now that people want
to use them there appears to be _some_ implementations (emphasis on
the word "some") that seem to have a problem processing them. Frankly,
that is a shortcoming of implementation, not of the protocol. The only
protocol shortcoming that has thus far been identified in the IOAM
discussion is the fact that EH don't exist in IPv4 and there is really
no alternative that provides the necessary functionality-- this is
something I believe IETF can work on.

Tom

---

From: Jai Kumar <[email-masked]>
Date: Sat, 6 Apr 2019 14:05:52 -0700

Tom,

We are talking about nexthop I.e. packet forwarding not member link selection in load balancing.
Basic packet forwarding is broken with options.

-Jai

> On Apr 6, 2019, at 12:55 PM, Tom Herbert <[email-masked]> wrote:
>

---

From: Mark Smith <[email-masked]>
Date: Sun, 7 Apr 2019 08:36:02 +1000

> Basic packet forwarding is broken with options.

If I understand this statement correctly, it isn't basic forwarding if
options other than the Hop-by-Hop option is considered.

IPv6 forwarding is designed to only use the fields in the fixed IPv6
header (with one exception) - that is why those fields are not
optional.

The only IPv6 option/EH that can be used in forwarding is the
Hop-by-Hop option, and it is required to be directly after the IPv6
fixed header.

Any other IPv6 options/EH are not designed nor intended to be used in
IPv6 forwarding.

Perhaps one thing people suggesting things to this Working Group can
forget is that our protocol is more than two decades old and that we
have a very large installed base of devices that support it and that
are being used constantly (this email is coming over IPv6).

So we can't just make arbitrary changes to the protocol to suit new
requirements, unlike greenfield IETF Working Groups with no or a very
limited installed base can. We have to try our hardest to accommodate
those new requirements within the specifications and within existing
deployments.

So if a new method of doing something is proposed, we here are going
to be looking to see how it can be done using existing, proven and
well-known methods first (e.g, adding supplementary information to
packets via IPv6-in-IPv6 encapsulation, ECMP via Flow Label rather
than digging up the UDP/TCP headers.)

Regards,
Mark.

---

From: Tom Herbert <[email-masked]>
Date: Sat, 6 Apr 2019 15:56:47 -0700

> header (with one exception) - that is why those fields are not

Thanks for the explanation Mark. I would only add that RFC8200 essentially
makes that one exception (HBH options) optional to process.

Tom

> > >> Please read the IOAM draft again. There is a variable length metadata
> stack as well with trace option. Forwarding becomes completely
> indeterministic.
> > >
> > > Jai,
> > >
> > >> In short using options forwarding gets impacted for EH. You still
> want to push for it go by all means but don't expect a silicon vendor or
> deployment to agree for it.
> > >
> > >>> You are right. There are two main cases where accessibility of
> transport header becomes an issue. And that is mainly because of current
> generation of fixed pipeline cell based architecture parse depth (not so
> much an issue for high end NPU where next packet header bytes can be DMA’ed
> in, though it impacts the performance. For eg. Cisco QFP NPU, EZ Chip,
> Juniper’s MX 3D NPU, or Broadcom’s Jerricho NPU ). This was mentioned
> earlier by Mikael as well.
> > >>>
> > >>>
> > >>>
> > >>> Case 1. SRv6
> > >>>
> > >>> Typically NPUs provide capability that it will handle 3 or 4 EH or
> SRH without performance penalty. With the presence of IFA header this means
> that for eg there can be only 2 or 3 SRH or EH. This is same as in MPLS
> label stack depth handling (PUSH or POP at PE router). Given the evolution
> SRV6 and MPLS-SR kind of headers, silicon vendors have taken a notice of it
> and are evolving the silicon arch to handle more chained headers.
> > >>>
> > >>>
> > >>>
> > >>> IFA supports fragmentation of metadata stack and also postcard mode
> so beyond the transport header it’s not an issue.
> > >>>
> > >>>
> > >>>
> > >>> Just a quick feedback on using IPv6 options, inserting hop by hop
> options for data path processing pushes the EH further down and impacts the
> node’s forwarding function (say if it is not able to reach Destination EH)
> vs if IFA header is the last EH then atmost transit node do not perform
> metadata collection which can be identified.
> > >>
> > >>> Cc: "Pengshuping (Peng Shuping)" <[email-masked]>, 6man <
> > >>> Subject: Re: [ippm] draft-ioametal-ippm-6man-ioam-ipv6-deployment-00
> > >>> Cc: "Pengshuping (Peng Shuping)" <[email-masked]>, 6man <
> > >>> Subject: Re: [ippm] draft-ioametal-ippm-6man-ioam-ipv6-deployment-00
> > >>> Its all explained in the requirements section of the draft. IFA
> header is essentially treated as an EH both in IPv4 and IPv6 (very similar
> to AH/ESP EH).
> > >>>
> > >>> Just like in AH/ESP the IP protocol is copied in the AH header and
> IP protocol field is replaced in the IP header with the AH proto.
> > >>>
> > >>> - Note that all the OAM data need to be deleted as well so position
> need to be optimal for removal. We can always do tail stamping of the data
> but this puts considerable constrains on the cell based architecture
> > >>> I am happy to discuss with you in person and explain more but bottom
> line is that this proposal is what came out with MSDC discussions in light
> of shortcomings from other IETF proposals.
> > >>>
> > >>>
> > >>>
> > >>> As they say, you are free to run whatever you want in your
> proprietary network, but if you expect to create an interoperable,
> deployable protocoI and get a protocol number assignment, you'll need to
> work in ietf. That is going to require justifying the protocol even in
> light of shortcomings in existing protocoIs.
> > >>>
> > >>>
> > >>>
> > >>> Tom
> > >>>
> > >>>
> > >>>
> > >>>
> > >>> Regards,
> > >>> -Jai
> > >>>
> > >>>
> > >>>>> I am speaking based on the actual deployment not based on if all
> the traffic is/will move to IPSec tunnel.
> > >>>>>
> > >>>>>> Even if we use flow label for ECMP, lack of accessibility of L4
> information (for various services, most simple is ACL) makes the proposal
> pretty much unusable.
> > >>>>>>
> > >>>>>> NIC cards are not used pervasively for ACLs and other services, I
> believe iptables in kernel are good enough for host but that’s not
> acceptable in networking elements.
> > >>>>>>
> > >>>>>> -Jai
> > >>>>>>
> > >>>>>>> We are certainly not designing for the lowest capable solutions
> but exploring optimal solutions which could help to keep the forwarding
> performance while inserting the iOAM data with variable length.
> > >>>>>>
> > >>>>>>    Shuping,
> > >>>>>>
> > >>>>>>> In terms of the forwarding efficiency, we would all agree that a
> short and fixed header would be better than a variable header.
> > >>>>>>>
> > >>>>>>> Best regards
> > >>>>>>> Shuping
> > >>>>>>>
> > >>>>>>>
> > >>>>>>> 收件人： Pengshuping (Peng Shuping)<[email-masked]>;Frank
> Brockners (fbrockne)<[email-masked]>;Mikael Abrahamsson<
> [email-masked]>
> > >>>>>>> 抄送： C. M. Heard<[email-masked]>;6man WG<[email-masked]>;Mark
> Smith<[email-masked]>;ippm<[email-masked]>
> > >>>>>>> 主题： RE: draft-ioametal-ippm-6man-ioam-ipv6-deployment-00
> > >>>>>>> I think that the question is whether we design for the lowest
> capable solutions. I assume someone can find solutions with even lower
> amount of parsing.. The low end solutions for these capabilities may need
> to either adopt with new designs or use solutions such as what is suggested
> in this thread by Tom Herbert.
> > >>>>>>> Personally, I don't think we should go this way, but to design
> appropriate solutions, with appropriate layers in the headers. For example,
> prevent dependencies between "remote" headers.
> > >>>>>>>
> > >>>>>>> Thanks,
> > >>>>>>> Barak
> > >>>>>>>
> > >>>>>>> Cc: C. M. Heard <[email-masked]>; 6man WG <[email-masked]>;
> Mark Smith <[email-masked]>; [email-masked]
> > >>>>>>> Subject: [ippm] 答复:
> > >>>>>>> The parsing depth of the early chips is 96B/128B. For the new
> chips the parsing depth has been increased but still limited. So Mikael's
> concern makes sense especially in the tracing mode that the IOAM data
> fields are going to increase significantly along the path, which will even
> push the Routing Header out of the parsing depth of the chip. So it would
> make sense to split the IOAM data part from the IOAM header/instruction
> part, and place the IOAM data after the RH or even after the L4 header in
> order to be able to read the L4 information to enable ECMP, as stated in
> the draft
> > >>>>>>> 抄送: C. M. Heard <[email-masked]>; 6man WG <[email-masked]>;
> Mark Smith <[email-masked]>; [email-masked]
> > >>>>>>> 主题: Re: [ippm] draft-ioametal-ippm-6man-ioam-ipv6-deployment-00
> > >>>>>>> Cc: Mark Smith <[email-masked]>; C. M. Heard <
> [email-masked]>; 6man WG <[email-masked]>; [email-masked]
> > >>>>>>> Subject: RE: draft-ioametal-ippm-6man-ioam-ipv6-deployment-00
> > >>>>>>>> * What one could consider - and which is one suggested
> deployment
> > >>>>>>>> model
> > >>>>>>> Yes, I think this is the only way. Is there a risk that
> intermediate routers would not be IOAM capable? I think the C1 requirement
> is really really hard to fulfil and I'm also afraid that adding the IOAM
> header will actually make ECMP stop working on some platforms (because they
> would not have the capability to look deep enough into the packet to find
> L4 information, so it'll go back to 2 tuple ECMP instead of 5 tuple.
> > >>>>>>>
> > >>>>>>> .....FB2: Given that IOAM is a domain specific feature - I
> expect that folks initially start to use it in domains (like e.g. a DC)
> where all boxes are IOAM capable and can trace the packet appropriately -
> or per Tom's note, can be configured so that one would do ECMP based on
> addresses and flow-label.  There are chips with pretty deep parsing
> capability (up to a few hundred bytes).
> > >>>>>>>
> > >>>>>>> When it comes to "cause harm" I imagine there are (at least) two
> ways to cause harm, one is privacy/secrecy/security loss and the other one
> is actual operational loss.
> > >>>>>>>
> > >>>>>>> I know of bugs where labeled packets went the wrong way and
> caused them to be lost, I've also seen bugs where bugs caused traffic to
> "hop" between VRFs in MPLS VPN (or to "global" VRF). I don't have numbers
> on this though.
> > >>>>>>>
> > >>>>>>> Depending on the deployment scenario, it might make sense to
> make IOAM packets not valid for non-IOAM aware devices (basically encap
> entire packet/payload), but that might be a problem for intermediate
> non-IOAM nodes then. This would affect the ECMP requirement. I can see
> cases where one would operationally turn on IOAM for some customers traffic
> and then see the problem go away because now ECMP changed.
> > >>>>>>>
> > >>>>>>> How much has the silicon people been involved so far in the
> design of the headers? What is the current thinking of amount of data going
> into the IOAM header? Considering things like trace options etc it seems to
> me the header can grow quite large? To satisfy the ECMP requirement what
> about putting the IOAM information as a trailer behind the regular payload?
> Or is there a problem for the hw to manipulate information that far into
> the packet (I also imagine this will considerably lower the forwarding
> > >>>>>>> .....FB2: There are quite a few folks with hardware background
> that co-author the IOAM drafts. Parsing capability differs between chips,
> as does the ability to deal with new/flexible headers and the ability to
> modify data in the extension headers. So the unfortunate answer to that
> question is "it depends" (what information do you gather, how large is your
> domain, what are the capabilities of your hardware/software forwarder,
> etc.), but I do expect that for specific deployment domains (e.g. DC,
> Enterprise) - but also for overlay networks / VPNs - we'll see an
> increasing amount of chips that are well suited for processing large
> extension header.
> > >>>>>>>
> > >>>>>>> Cheers, Frank
> > >>>>>>>
> > >>>>>>> --