Subject: Re: [sfc] [ippm] encapsulation of IOAM data in various protocols - follow up from WG discussion in London

From: Greg Mirsky <[email-masked]>
Date: Thu, 12 Apr 2018 18:54:27 +0200

Hi Frank,
thank you for sharing your points. Please find my notes in-line and tagged
GIM>>. I believe that this is very much relevant to work of other working
groups that directly work on the overlay encapsulations in the center of
the discussion and hence I've added them to the list. Hope we'll have more
opinions to reach the conclusion that is acceptable to all.

Regards,
Greg

On Wed, Apr 11, 2018 at 12:02 PM, Frank Brockners (fbrockne) <
[email-masked]> wrote:

> Back at the IPPM meeting in London, we discussed several drafts dealing
> with the encapsulation of IOAM data in various protocols
> (draft-brockners-ippm-ioam-vxlan-gpe-00, draft-brockners-ippm-ioam-geneve-00,
> draft-weis-ippm-ioam-gre-00). One discussion topic that we decided to take
> to the list was the question on whether draft-ooamdt-rtgwg-ooam-header
> could be leveraged.  After carefully considering draft-ooamdt-rtgwg-ooam-header,
> I came to the conclusion that the “OOAM header” does not meet the needs of
> IOAM:
>
> * Efficiency: IOAM adds data to live user traffic. As such, an
> encapsulation needs to be as efficient as possible. The “OOAM header” is 8
> bytes long. The approach for IOAM data encapsulation in the above mentioned
> drafts only requires 4 bytes. Using the OOAM header approach would add an
> unnecessary overhead of 4 bytes – which is significant.
>
GIM>> The difference in four octets is because OOAM Header:

   - provides more flexibility, e.g. Flags field and Reserved fields;
   - supports larger OAM packets than iOAM header;
   - is future proof by supporting versioning (Version field).

* Maturity: IOAM has several implementations, which were also shown at
> recent IETF hackathons – and we’re expecting additional implementations to
> be publicized soon. Interoperable implementations need timely
> specifications. Despite the question being asked, the recent thread on OOAM
> in the NVO3 list hasn’t revealed any implementation of the OOAM header. In
> addition, the thread revealed that several fundamental questions about the
> OOAM header are still open, such as whether or how active OAM mechanisms
> within protocols such as Geneve would apply to the OOAM header. This
> ultimately means that we won’t get to a timely specification.
>
GIM>> May I ask which encapsulations supported by the implementations you
refer to. Until very recently all iOAM proposals were to use meta-data TLV
in, e.g. Geneve and NSH. And if these or some of these implementations
already updated to the newly proposed iOAM shim, I don't see problem in
making them use OOAM Header. Would you agree?

> * Scope: It isn’t entirely clear to which protocols the OOAM header would
> ultimately apply to. The way the OOAM header is defined, OOAM uses a 8-bit
> field for “Next Prot”, the next protocol. Some protocols that IOAM data
> needs to be encapsulated into use 16-bits for their next protocol code
> points. See e.g. the GRE encapsulation – as specified in
> draft-weis-ippm-ioam-gre-00.
>
GIM>> The first paragraph of the Introduction section states:
   New protocols that support overlay networks like VxLAN-GPE
   [I-D.ietf-nvo3-vxlan-gpe], GUE [I-D.ietf-nvo3-gue], Geneve
   [I-D.ietf-nvo3-geneve], BIER [I-D.ietf-bier-mpls-encapsulation], and
   NSH [I-D.ietf-sfc-nsh] support multi-protocol payload, e.g.
   Ethernet, IPv4/IPv6, and recognize Operations, Administration, and
   Maintenance (OAM) as one of distinct types.  That ensures that
   Overlay OAM (OOAM)packets are sharing fate with Overlay data packet
   traversing the underlay.
I'm updating the OOAM Header draft and along with cleaning nits will update
reference to GUE. I think that the list and the statemnt are quite clear in
identifying the scope of networks that may benefit from using not only
common OOAM Header but common OOAM mechanisms, e.g. Echo Request/Reply
<https://tools.ietf.org/html/draft-ooamdt-rtgwg-demand-cc-cv-03>.

With the above in mind, I’d suggest that the WG moves forward with specific
> definitions for encapsulating IOAM data into protocols – per the above
> mentioned drafts.
>
>
>
> Regards, Frank
>
> _______________________________________________
> ippm mailing list
> [email-masked]
> https://www.ietf.org/mailman/listinfo/ippm
>
>

---

From: "Frank Brockners (fbrockne)" <[email-masked]>
Date: Thu, 12 Apr 2018 18:06:21 +0000

Hi Greg,

thanks – and it seems that we’re on the same page with regards to efficiency (4 bytes of non-required overhead) and maturity (or lack of) of OOAM.

On the IOAM implementation: There are several implementations of IOAM. Some of which have recently been worked on and shown at an IETF hackathon, see https://datatracker.ietf.org/meeting/100/materials/slides-100-hackathon-sessa-in-situ-oam-ioam - where we’ve shown IPv6 and VXLAN-GPE with IOAM – on FD.io/VPP as well as on Barefoot Tofino. You probably also remember the Netronome/Broadcom demo - https://www.youtube.com/watch?v=j9FbD4a3F4E .
Below you seem to be specifically referring to the IOAM open source implementation in FD.io/VPP: There are protocol encapsulations for VXLAN-GPE, NSH, and IPv6 implemented in FD.io/VPP. The current code uses the “next header approach” for VXLAN-GPE and it leverages MD-Type 2 for NSH. As you’re well aware, there the discussion in SFC whether to use MD-Type 2 or next header encapsulating IOAM data in NSH isn’t yet settled, hence we’ll refrain from updating the code until SFC WG has come to a conclusion.

Could you provide a pointer to an OOAM implementation?

Thanks,
Frank

From: Greg Mirsky <[email-masked]>
Sent: Donnerstag, 12. April 2018 18:54
To: Frank Brockners (fbrockne) <[email-masked]>
Cc: IETF IPPM WG <[email-masked]>; NVO3 <[email-masked]>; Service Function Chaining IETF list <[email-masked]>; [email-masked]
Subject: Re: [ippm] encapsulation of IOAM data in various protocols - follow up from WG discussion in London

Hi Frank,
thank you for sharing your points. Please find my notes in-line and tagged GIM>>. I believe that this is very much relevant to work of other working groups that directly work on the overlay encapsulations in the center of the discussion and hence I've added them to the list. Hope we'll have more opinions to reach the conclusion that is acceptable to all.

Regards,
Greg

On Wed, Apr 11, 2018 at 12:02 PM, Frank Brockners (fbrockne) <[email-masked]<mailto:[email-masked]>> wrote:
Back at the IPPM meeting in London, we discussed several drafts dealing with the encapsulation of IOAM data in various protocols (draft-brockners-ippm-ioam-vxlan-gpe-00, draft-brockners-ippm-ioam-geneve-00, draft-weis-ippm-ioam-gre-00). One discussion topic that we decided to take to the list was the question on whether draft-ooamdt-rtgwg-ooam-header could be leveraged.  After carefully considering draft-ooamdt-rtgwg-ooam-header, I came to the conclusion that the “OOAM header” does not meet the needs of IOAM:
* Efficiency: IOAM adds data to live user traffic. As such, an encapsulation needs to be as efficient as possible. The “OOAM header” is 8 bytes long. The approach for IOAM data encapsulation in the above mentioned drafts only requires 4 bytes. Using the OOAM header approach would add an unnecessary overhead of 4 bytes – which is significant.
GIM>> The difference in four octets is because OOAM Header:

  *   provides more flexibility, e.g. Flags field and Reserved fields;
  *   supports larger OAM packets than iOAM header;
  *   is future proof by supporting versioning (Version field).
* Maturity: IOAM has several implementations, which were also shown at recent IETF hackathons – and we’re expecting additional implementations to be publicized soon. Interoperable implementations need timely specifications. Despite the question being asked, the recent thread on OOAM in the NVO3 list hasn’t revealed any implementation of the OOAM header. In addition, the thread revealed that several fundamental questions about the OOAM header are still open, such as whether or how active OAM mechanisms within protocols such as Geneve would apply to the OOAM header. This ultimately means that we won’t get to a timely specification.
GIM>> May I ask which encapsulations supported by the implementations you refer to. Until very recently all iOAM proposals were to use meta-data TLV in, e.g. Geneve and NSH. And if these or some of these implementations already updated to the newly proposed iOAM shim, I don't see problem in making them use OOAM Header. Would you agree?

* Scope: It isn’t entirely clear to which protocols the OOAM header would ultimately apply to. The way the OOAM header is defined, OOAM uses a 8-bit field for “Next Prot”, the next protocol. Some protocols that IOAM data needs to be encapsulated into use 16-bits for their next protocol code points. See e.g. the GRE encapsulation – as specified in draft-weis-ippm-ioam-gre-00.
I'm updating the OOAM Header draft and along with cleaning nits will update reference to GUE. I think that the list and the statemnt are quite clear in identifying the scope of networks that may benefit from using not only common OOAM Header but common OOAM mechanisms, e.g. Echo Request/Reply<https://tools.ietf.org/html/draft-ooamdt-rtgwg-demand-cc-cv-03>.

With the above in mind, I’d suggest that the WG moves forward with specific definitions for encapsulating IOAM data into protocols – per the above mentioned drafts.

Regards, Frank

_______________________________________________
ippm mailing list
[email-masked]<mailto:[email-masked]>
https://www.ietf.org/mailman/listinfo/ippm

---

From: Tom Herbert <[email-masked]>
Date: Thu, 12 Apr 2018 11:37:43 -0700

On Thu, Apr 12, 2018 at 9:54 AM, Greg Mirsky <[email-masked]> wrote:
> groups that directly work on the overlay encapsulations in the center of the
> discussion and hence I've added them to the list. Hope we'll have more
> On Wed, Apr 11, 2018 at 12:02 PM, Frank Brockners (fbrockne)
> <[email-masked]> wrote:
>>
>> (draft-brockners-ippm-ioam-vxlan-gpe-00,
>> draft-brockners-ippm-ioam-geneve-00, draft-weis-ippm-ioam-gre-00). One
>> discussion topic that we decided to take to the list was the question on
>> whether draft-ooamdt-rtgwg-ooam-header could be leveraged.  After carefully
>> considering draft-ooamdt-rtgwg-ooam-header, I came to the conclusion that
>> the “OOAM header” does not meet the needs of IOAM:
>>

What is the relationship between this proposal and
draft-fioccola-v6ops-ipv6-alt-mark which is currently searching for a
way to squeeze out two bits in the IP header for performance
measurement?

Tom

> provides more flexibility, e.g. Flags field and Reserved fields;
> supports larger OAM packets than iOAM header;
> is future proof by supporting versioning (Version field).
>>
> common OOAM Header but common OOAM mechanisms, e.g. Echo Request/Reply.
>
>> With the above in mind, I’d suggest that the WG moves forward with
>> specific definitions for encapsulating IOAM data into protocols – per the
>> above mentioned drafts.
>>
>>
>>
>> Regards, Frank
>>
>>
> Int-area mailing list
> [email-masked]
> https://www.ietf.org/mailman/listinfo/int-area
>

---

From: Greg Mirsky <[email-masked]>
Date: Thu, 12 Apr 2018 23:28:31 +0200

Hi Tom,
I think you refer to the proposal on how to apply RFC 8321
<https://tools.ietf.org/html/rfc8321> in IPv6 networks. Using two bits-long
field for Alternate Marking is just one option as you can find in the draft
on compact Alt.marking
<https://tools.ietf.org/html/draft-mizrahi-ippm-compact-alternate-marking-01>.
But to answer your question, there's no connection or dependency between
the Alternate Marking method and OOAM. Will note that we have documents on
the applicability of the Alternate Marking method in BIER, SFC NSH, and
Geneve.

Regards,
Greg

On Thu, Apr 12, 2018 at 8:37 PM, Tom Herbert <[email-masked]> wrote:

> the

---

From: Greg Mirsky <[email-masked]>
Date: Thu, 12 Apr 2018 23:35:20 +0200

Hi Frank,
I think you've misunderstood my response to your statements:

   - the scope of OOAM, contrary to what you've stated, is clearly stated
   in the draft;
   - what you present as "efficiency" I consider to be serious limitations
   (lack of versioning, limited size for data, and no future extension) that
   should be explained and thoroughly discussed by the WGs that develop
   corresponding overlay networks before IPPM WG makes any decision.

Regards,
Greg

On Thu, Apr 12, 2018 at 8:06 PM, Frank Brockners (fbrockne) <
[email-masked]> wrote:

> Hi Greg,
>
>
>
> thanks – and it seems that we’re on the same page with regards to
> efficiency (4 bytes of non-required overhead) and maturity (or lack of) of
> OOAM.
>
>
>
> On the IOAM implementation: There are several implementations of IOAM.
> Some of which have recently been worked on and shown at an IETF hackathon,
> see https://datatracker.ietf.org/meeting/100/materials/slides-
> 100-hackathon-sessa-in-situ-oam-ioam - where we’ve shown IPv6 and
> VXLAN-GPE with IOAM – on FD.io/VPP as well as on Barefoot Tofino. You
> probably also remember the Netronome/Broadcom demo -
> https://www.youtube.com/watch?v=j9FbD4a3F4E .
>
> Below you seem to be specifically referring to the IOAM open source
> implementation in FD.io/VPP: There are protocol encapsulations for
> VXLAN-GPE, NSH, and IPv6 implemented in FD.io/VPP. The current code uses
> the “next header approach” for VXLAN-GPE and it leverages MD-Type 2 for
> NSH. As you’re well aware, there the discussion in SFC whether to use
> MD-Type 2 or next header encapsulating IOAM data in NSH isn’t yet settled,
> hence we’ll refrain from updating the code until SFC WG has come to a
> conclusion.
>
>
>
> *From:* Greg Mirsky <[email-masked]>
> *Sent:* Donnerstag, 12. April 2018 18:54
> *To:* Frank Brockners (fbrockne) <[email-masked]>
> *Cc:* IETF IPPM WG <[email-masked]>; NVO3 <[email-masked]>; Service
> Function Chaining IETF list <[email-masked]>; [email-masked]
> *Subject:* Re: [ippm] encapsulation of IOAM data in various protocols -
> follow up from WG discussion in London
>
>
>
> Hi Frank,
>
> I'm updating the OOAM Header draft and along with cleaning nits will
> update reference to GUE. I think that the list and the statemnt are quite
> clear in identifying the scope of networks that may benefit from using not
> only common OOAM Header but common OOAM mechanisms, e.g. Echo
> Request/Reply

---

From: Tom Herbert <[email-masked]>
Date: Thu, 12 Apr 2018 14:46:48 -0700

>> unnecessary overhead of 4 bytes – which is significant.
Greg,

I'm missing something here. I looked at the drafts you referenced and
each of them looks like the overhead for OAM is greater that four
bytes. In each there is some overhead equivalent to type/length, for
instance in Geneve four bytes are needed for option class, type, and
length. Unless the the OAM data is zero length, I don't see how this
adds up to only four bytes of overhead.

Tom

---

From: Greg Mirsky <[email-masked]>
Date: Thu, 12 Apr 2018 23:50:33 +0200

Hi Tom,
could you please mention which drafts, iOAM or OOAM, you refer to. Please
note, that OOAM supports both active and hybrid OAM methods, while iOAM
only the latter.

Regards,
Greg

On Thu, Apr 12, 2018 at 11:46 PM, Tom Herbert <[email-masked]> wrote:

> update

---

From: Tom Herbert <[email-masked]>
Date: Thu, 12 Apr 2018 14:53:16 -0700

On Thu, Apr 12, 2018 at 2:50 PM, Greg Mirsky <[email-masked]> wrote:
> note, that OOAM supports both active and hybrid OAM methods, while iOAM only
> the latter.

Section 3 of draft-brockners-ippm-ioam-geneve-00 for instance.

---

From: Greg Mirsky <[email-masked]>
Date: Thu, 12 Apr 2018 23:56:12 +0200

Hi Tom,
I'll let Frank answer your question as it is on iOAM, not OOAM.

Regards,
Greg

On Thu, Apr 12, 2018 at 11:53 PM, Tom Herbert <[email-masked]> wrote:

> only
> >> >> drafts only requires 4 bytes. Using the OOAM header approach would
> add
> >> >> an
> >> >> recent IETF hackathons – and we’re expecting additional
> implementations
> >> >> to
> >> >> specifications. Despite the question being asked, the recent thread
> on
> >> >> OOAM
> >> >> in the NVO3 list hasn’t revealed any implementation of the OOAM
> header.
> >> >> In
> >> >> addition, the thread revealed that several fundamental questions
> about
> >> >> the
> >> > reference to GUE. I think that the list and the statemnt are quite
> clear
> >> > in

---

From: Mickey Spiegel <[email-masked]>
Date: Thu, 12 Apr 2018 15:31:23 -0700

Tom,

> adds up to only four bytes of overhead.

The four versus eight bytes just refers to the fields in the four bytes of
IOAM
info, that is common to all IOAM options. Beyond that, there are IOAM option
specific fields. For example if doing one of the IOAM trace options, there
are
four bytes of trace option header, including the IOAM-trace-type, NodeLen,
Flags, and RemainingLen fields. These are followed by the node data list
containing the per hop IOAM information.

In looking at the OOAM header content, it has nothing to do with any of the
IOAM information after the first four bytes. It contains another variant of
the
information in the first four bytes of IOAM info, spread out over eight
bytes.

> > provides more flexibility, e.g. Flags field and Reserved fields;

The flags field only has one defined flag at the moment, for a timestamp
block. For IOAM trace we need per hop timestamps, which the timestamp
block cannot address, i.e. the timestamp block is redundant for IOAM.

> > supports larger OAM packets than iOAM header;

For IOAM purposes, 1020 octets is more than enough.

> > is future proof by supporting versioning (Version field).

IMO, taking the first two bits of the IOAM-Type to define a Version field
would be a good thing. This does not require adding four more bytes of
overhead. 64 IOAM-Types is more than enough.

---

From: "Carlos Pignataro (cpignata)" <[email-masked]>
Date: Fri, 13 Apr 2018 01:37:50 +0000

Hi, Greg,

Thanks for the quick response — Frank provided answers to your points, but I do have one question about your response (and I will squeeze in a couple of additional comments).

Please see inline.

On Apr 12, 2018, at 12:54 PM, Greg Mirsky <[email-masked]<mailto:[email-masked]>> wrote:

Hi Frank,

Regards,
Greg

Comment: engineering is usually about making trade-offs. Having Reserved fields, unnecessary (and over-generalized) Flags, and a Version field for this extra OOAM header does not seem to justify the ROI in adding an extra header and extra parsing. This goes back to ‘what problem?'. Having a lot of extra fields to parse adds overhead, not only in size, but also in complexity tax. Please see Fundamental Truths (6a), (10), and (12) [RFC 1925].
* Maturity: IOAM has several implementations, which were also shown at recent IETF hackathons – and we’re expecting additional implementations to be publicized soon. Interoperable implementations need timely specifications. Despite the question being asked, the recent thread on OOAM in the NVO3 list hasn’t revealed any implementation of the OOAM header. In addition, the thread revealed that several fundamental questions about the OOAM header are still open, such as whether or how active OAM mechanisms within protocols such as Geneve would apply to the OOAM header. This ultimately means that we won’t get to a timely specification.
GIM>> May I ask which encapsulations supported by the implementations you refer to. Until very recently all iOAM proposals were to use meta-data TLV in, e.g. Geneve and NSH. And if these or some of these implementations already updated to the newly proposed iOAM shim, I don't see problem in making them use OOAM Header. Would you agree?

Question: The “like” in the first sentence, basically implies “not all encapsulations”. Therefore, https://xkcd.com/927/.

Further, what are “protocols that support overlay networks”, and how are you choosing your examples?

Question: I do not think this is quite clear as you state, given the fact that the word “like” adds imprecision and ambiguity. IP-in-IP? L2TPv3? Etc?

Thank you for considering these questions!
With the above in mind, I’d suggest that the WG moves forward with specific definitions for encapsulating IOAM data into protocols – per the above mentioned drafts.

Regards, Frank

_______________________________________________
Int-area mailing list
[email-masked]<mailto:[email-masked]>
https://www.ietf.org/mailman/listinfo/int-area

Thanks,

— Carlos Pignataro

---

From: "Carlos Pignataro (cpignata)" <[email-masked]>
Date: Fri, 13 Apr 2018 01:56:12 +0000

Hi Greg,

Thanks for engaging and for your interest. Please find a couple of follow-ups to your comments, consolidating from the plurality of emails you sent with your responses.

The comparison you are suggestion between IOAM and OOAM is a red-herring, and as such I’ll go quiet and not respond on this thread after this email.

But for completeness:

1. You did not answer this, and again selectively skipped it. Can you please provide and answer?
Could you provide a pointer to an OOAM implementation?

2. You wrote:

Please note, that OOAM supports both active and hybrid OAM methods, while iOAM only the latter.

You are comparing apples with oranges. IOAM *is* a hybrid OAM method. OOAM is neither. It “supports” means that by itself it provides no value.

(And by the way, why Timestamps on this header when the OAM Packet itself will have timestamps if needed?!)

3. You wrote:

  *   the scope of OOAM, contrary to what you've stated, is clearly stated in the draft;

But the scope is not clear, to me at least, since it has no precision (what is included, what is excluded)? Is MPLS an overlay protocol? Is SRv6?

4. You wrote:

  *   what you present as "efficiency" I consider to be serious limitations (lack of versioning, limited size for data, and no future extension)

I do not believe this is an accurate characterization… BFD has less extensibility and more limitations than IOAM, and it’s great because it serves its function.

  *   that should be explained and thoroughly discussed by the WGs that develop corresponding overlay networks before IPPM WG makes any decision.

This is not an “overlay network” issue from the IPPM scope on the protocol. Let’s have a technical discussion and not artificially attempt to slow things down.

Many Thanks,

— Carlos Pignataro

On Apr 12, 2018, at 5:35 PM, Greg Mirsky <[email-masked]<mailto:[email-masked]>> wrote:

Hi Frank,

  *   the scope of OOAM, contrary to what you've stated, is clearly stated in the draft;
  *   what you present as "efficiency" I consider to be serious limitations (lack of versioning, limited size for data, and no future extension) that should be explained and thoroughly discussed by the WGs that develop corresponding overlay networks before IPPM WG makes any decision.

Regards,
Greg

On Thu, Apr 12, 2018 at 8:06 PM, Frank Brockners (fbrockne) <[email-masked]<mailto:[email-masked]>> wrote:
Hi Greg,

On the IOAM implementation: There are several implementations of IOAM. Some of which have recently been worked on and shown at an IETF hackathon, see https://datatracker.ietf.org/meeting/100/materials/slides-100-hackathon-sessa-in-situ-oam-ioam - where we’ve shown IPv6 and VXLAN-GPE with IOAM – on FD.io/VPP<http://FD.io/VPP> as well as on Barefoot Tofino. You probably also remember the Netronome/Broadcom demo - https://www.youtube.com/watch?v=j9FbD4a3F4E .
Below you seem to be specifically referring to the IOAM open source implementation in FD.io/VPP:<http://FD.io/VPP:> There are protocol encapsulations for VXLAN-GPE, NSH, and IPv6 implemented in FD.io/VPP<http://FD.io/VPP>. The current code uses the “next header approach” for VXLAN-GPE and it leverages MD-Type 2 for NSH. As you’re well aware, there the discussion in SFC whether to use MD-Type 2 or next header encapsulating IOAM data in NSH isn’t yet settled, hence we’ll refrain from updating the code until SFC WG has come to a conclusion.

Thanks,
Frank

From: Greg Mirsky <[email-masked]<mailto:[email-masked]>>
Sent: Donnerstag, 12. April 2018 18:54
To: Frank Brockners (fbrockne) <[email-masked]<mailto:[email-masked]>>
Cc: IETF IPPM WG <[email-masked]<mailto:[email-masked]>>; NVO3 <[email-masked]<mailto:[email-masked]>>; Service Function Chaining IETF list <[email-masked]<mailto:[email-masked]>>; [email-masked]<mailto:[email-masked]>
Subject: Re: [ippm] encapsulation of IOAM data in various protocols - follow up from WG discussion in London

Hi Frank,

Regards,
Greg

Regards, Frank

_______________________________________________
nvo3 mailing list
[email-masked]<mailto:[email-masked]>
https://www.ietf.org/mailman/listinfo/nvo3

---

From: Tom Herbert <[email-masked]>
Date: Thu, 12 Apr 2018 18:59:38 -0700

> containing the per hop IOAM information.
All of that and people are worried about having four extra bytes of
overhead? :-)

How big are these encapsulation headers in data packets going to be?

Tom

---

From: Tom Herbert <[email-masked]>
Date: Thu, 12 Apr 2018 22:17:21 -0700

Mickey,

Looking at these ippm drafts more closely, I have a much more
fundamental concern.

In draft-brockners-ippm-ioam-geneve-00 for instance, there is the text
in the introduction:

"In-situ OAM (IOAM) records OAM information within the packet while
the packet traverses a particular network domain.  The term "in-situ"
refers to the fact that the IOAM data fields are added to the data
packets rather than is being sent within packets specifically
dedicated to OAM.  This document defines how IOAM data fields are
transported as part of the Geneve [I-D.ietf-nvo3-geneve]
encapsulation."

I assume this means that as packets with Geneve encapsulation traverse
the network they are interpreted by intermediate nodes as being
Geneve. Since Geneve is a UDP encapsulation, then the destination UDP
port number would be used to identify packets as being Geneve. So an
intermediate device might be looking for UDP packets destined to port
6081 (the assigned UDP port for Geneve). If my understanding is
correct, then this is a problem.

UDP port numbers do not have global meaning. An intermediate device
may very well see UDP packets destined to port 6081 that are not
actually Geneve. This scenario is discussed in RFC7605:

"...intermediate device interprets traffic based on the port number.
It is important to recognize that any interpretation of port numbers
-- except at the endpoints -- may be incorrect, because port numbers
are meaningful only at the endpoints."

If the UDP data is modified, as the draft would imply, then
misinterpretation may also mean silent data corruption of packets. A
protocol that would allow this seems pretty incorrect! Note that this
would be true also for any UDP encapsulation that the network tries to
interpret.

I am also wondering if hop-by-hop options been considered for this
application? Their interpretation in the network is unabiguous and
they also have the advantage that the work with any IP protocol or
encapsulation.

Thanks,
Tom

---

From: "Frank Brockners (fbrockne)" <[email-masked]>
Date: Fri, 13 Apr 2018 06:28:57 +0000

Tom,

the term "overhead" here refers to the number of extra bytes used in the parent protocol to carry IOAM data. IOAM data itself is of course not counted for the comparison, because it would need to be carried in both cases.

Using your Geneve reference as an example, in order to carry IOAM data, we need to add another Geneve Option class along with type fields etc. per the Geneve draft spec - which sums up to 4 bytes:

    0                   1                   2                   3
    0 1 2 3 4 5 6 7 8 9 0 1 2 3 4 5 6 7 8 9 0 1 2 3 4 5 6 7 8 9 0 1
   |  Option Class  =  TBD_IOAM    |     Type      |R|R|R| Length  | 

The OOAM header requires 8 bytes to serve the very same purpose:

    0                   1                   2                   3
    0 1 2 3 4 5 6 7 8 9 0 1 2 3 4 5 6 7 8 9 0 1 2 3 4 5 6 7 8 9 0 1
   +-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+
   | V |           Msg Type        |           Length              |
   +-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+
   |             Flags             |    Reserved   |   Next Prot   |
   +-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+

HTH.

Frank

-----Original Message-----
From: Tom Herbert <[email-masked]> 
Sent: Donnerstag, 12. April 2018 23:53
To: Greg Mirsky <[email-masked]>
Cc: Frank Brockners (fbrockne) <[email-masked]>; NVO3 <[email-masked]>; [email-masked]; Service Function Chaining IETF list <[email-masked]>; IETF IPPM WG <[email-masked]>
Subject: Re: [Int-area] [ippm] encapsulation of IOAM data in various protocols - follow up from WG discussion in London

On Thu, Apr 12, 2018 at 2:50 PM, Greg Mirsky <[email-masked]> wrote:
> Hi Tom,
> could you please mention which drafts, iOAM or OOAM, you refer to. 

>> > groups that directly work on the overlay encapsulations in the 
>> > center of the discussion and hence I've added them to the list. 
>> > Hope we'll have more opinions to reach the conclusion that is 
>> > acceptable to all.
>> >
>> > Regards,
>> > Greg
>> >
>> >> Back at the IPPM meeting in London, we discussed several drafts 
>> >> dealing with the encapsulation of IOAM data in various protocols 
>> >> draft-brockners-ippm-ioam-geneve-00, draft-weis-ippm-ioam-gre-00). 
>> >> One discussion topic that we decided to take to the list was the 
>> >> question on whether draft-ooamdt-rtgwg-ooam-header could be 
>> >> leveraged.  After carefully considering 
>> >> draft-ooamdt-rtgwg-ooam-header, I came to the conclusion that the 
>> >> “OOAM header” does not meet the needs of IOAM:
>> >>
>> >> bytes long. The approach for IOAM data encapsulation in the above 
>> >> mentioned drafts only requires 4 bytes. Using the OOAM header 
>> >> approach would add an unnecessary overhead of 4 bytes – which is 
>> >> significant.
>> Greg,
>>
>> > supports larger OAM packets than iOAM header; is future proof by 
>> > supporting versioning (Version field).
>> >>
>> >> * Maturity: IOAM has several implementations, which were also 
>> >> shown at recent IETF hackathons – and we’re expecting additional 
>> >> implementations to be publicized soon. Interoperable 
>> >> implementations need timely specifications. Despite the question 
>> >> being asked, the recent thread on OOAM in the NVO3 list hasn’t 
>> >> revealed any implementation of the OOAM header.
>> >> In
>> >> about the OOAM header are still open, such as whether or how 
>> >> active OAM mechanisms within protocols such as Geneve would apply 
>> >> to the OOAM header. This ultimately means that we won’t get to a 
>> >> timely specification.
>> >
>> > GIM>> May I ask which encapsulations supported by the 
>> > GIM>> implementations
>> > you
>> > refer to. Until very recently all iOAM proposals were to use 
>> > meta-data TLV in, e.g. Geneve and NSH. And if these or some of 
>> > these implementations already updated to the newly proposed iOAM 
>> > shim, I don't see problem in making them use OOAM Header. Would you 
>> > agree?
>> >
>> >>
>> >> * Scope: It isn’t entirely clear to which protocols the OOAM 
>> >> header would ultimately apply to. The way the OOAM header is 
>> >> defined, OOAM uses a 8-bit field for “Next Prot”, the next 
>> >> protocol. Some protocols that IOAM data needs to be encapsulated 
>> >> into use 16-bits for their next protocol code points. See e.g. the 
>> >> GRE encapsulation – as specified in draft-weis-ippm-ioam-gre-00.
>> >
>> > I'm updating the OOAM Header draft and along with cleaning nits 
>> > will update reference to GUE. I think that the list and the 
>> > statemnt are quite clear in identifying the scope of networks that 
>> > may benefit from using not only common OOAM Header but common OOAM 
>> > mechanisms, e.g. Echo Request/Reply.
>> >
>> >> specific definitions for encapsulating IOAM data into protocols – 
>> >> per the above mentioned drafts.
>> >>
>> >>
>> >>
>> >> Regards, Frank
>> >>
>> >>

---

From: "徐小虎(义先)" <[email-masked]>
Date: Fri, 13 Apr 2018 15:42:23 +0800

Hi,
It said in draft-brockners-ippm-ioam-vxlan-gpe-00:
"   [I-D.ietf-nvo3-vxlan-gpe] defines an "O bit" for OAM packets.  Per
   [I-D.ietf-nvo3-vxlan-gpe] the O bit indicates that the packet
   contains an OAM message instead of data payload.  Packets that carry
   IOAM data fields in addition to regular data payload / customer
   traffic must not set the O bit.  Packets that carry only IOAM data
   fields without any payload must set the O bit."
My first question is: if the Next Protocol field within the VXLAN-GPE header should be resorted to indicate the IOAM, why do we still need the "O" bit? 
It said in draft-brockners-ippm-ioam-vxlan-gpe-00:
"Next Protocol:  8-bit unsigned integer that determines the type of
      header following IOAM protocol.  The value is from the IANA             registry setup for VXLAN GPE Next Protocol defined in
      [I-D.ietf-nvo3-vxlan-gpe]."
My second question is: why the "Next Protocol" is designed to be context-specific (i.e., specific to the tunnel over which the IOAM data fields are contained). In other words, wouldn't it be better to make the Next Protocol tunnel-independant since the IOAM is intended to be added into various tunnel encapsulations?

My third question is: does it means intermediate nodes must be aware of various tunnel encapsulations since the IOAM data field is behind the tunnel header? wouldn't it be better to carry the IOAM data just behind the outer IP header?
Best regards,Xiaohu
On Wed, Apr 11, 2018 at 12:02 PM, Frank Brockners (fbrockne) <[email-masked]> wrote:
Back at the IPPM meeting in London, we discussed several drafts dealing with the encapsulation of IOAM data in various protocols (draft-brockners-ippm-ioam-vxlan-gpe-00, draft-brockners-ippm-ioam-geneve-00, draft-weis-ippm-ioam-gre-00).
 One discussion topic that we decided to take to the list was the question on whether draft-ooamdt-rtgwg-ooam-header could be leveraged.  After carefully considering draft-ooamdt-rtgwg-ooam-header, I came to the conclusion that the “OOAM header” does not meet
 the needs of IOAM:* Efficiency: IOAM adds data to live user traffic. As such, an encapsulation needs to be as efficient as possible. The “OOAM header” is 8 bytes long. The approach for IOAM data encapsulation in the above mentioned drafts
 only requires 4 bytes. Using the OOAM header approach would add an unnecessary overhead of 4 bytes – which is significant.GIM>> The difference in four octets is because OOAM Header:provides more flexibility, e.g. Flags field and Reserved fields;supports larger OAM packets than iOAM header;is future proof by supporting versioning (Version field).
* Maturity: IOAM has several implementations, which were also shown at recent IETF hackathons – and we’re expecting additional implementations to be publicized soon. Interoperable implementations need timely specifications.
 Despite the question being asked, the recent thread on OOAM in the NVO3 list hasn’t revealed any implementation of the OOAM header. In addition, the thread revealed that several fundamental questions about the OOAM header are still open, such as whether or
 how active OAM mechanisms within protocols such as Geneve would apply to the OOAM header. This ultimately means that we won’t get to a timely specification.GIM>> May I ask which encapsulations supported by the implementations you refer to. Until very recently all iOAM proposals were to use meta-data TLV in, e.g. Geneve and NSH. And if these or some of these implementations already updated to the newly proposed iOAM shim, I don't see problem in making them use OOAM Header. Would you agree? * Scope: It isn’t entirely clear to which protocols the OOAM header would ultimately apply to. The way the OOAM header is defined, OOAM uses a 8-bit field for “Next Prot”, the next protocol. Some protocols that IOAM data
 needs to be encapsulated into use 16-bits for their next protocol code points. See e.g. the GRE encapsulation – as specified in draft-weis-ippm-ioam-gre-00.GIM>> The first paragraph of the Introduction section states:   New protocols that support overlay networks like VxLAN-GPE   [I-D.ietf-nvo3-vxlan-gpe], GUE [I-D.ietf-nvo3-gue], Geneve   [I-D.ietf-nvo3-geneve], BIER [I-D.ietf-bier-mpls-encapsulation], and   NSH [I-D.ietf-sfc-nsh] support multi-protocol payload, e.g.   Ethernet, IPv4/IPv6, and recognize Operations, Administration, and   Maintenance (OAM) as one of distinct types.  That ensures that   Overlay OAM (OOAM)packets are sharing fate with Overlay data packet   traversing the underlay. I'm updating the OOAM Header draft and along with cleaning nits will update reference to GUE. I think that the list and the statemnt are quite clear in identifying the scope of networks that may benefit from using not only common OOAM Header but common OOAM mechanisms, e.g. Echo Request/Reply.
With the above in mind, I’d suggest that the WG moves forward with specific definitions for encapsulating IOAM data into protocols – per the above mentioned drafts. Regards, Frank
_______________________________________________

ippm mailing list
[email-masked]

---

From: "Frank Brockners (fbrockne)" <[email-masked]>
Date: Fri, 13 Apr 2018 08:59:35 +0000

Hi Xiaohu,

please see inline (“...FB“)

From: nvo3 <[email-masked]> On Behalf Of ???(??)
Sent: Freitag, 13. April 2018 09:42
To: Int-area <[email-masked]>; Frank Brockners (fbrockne) <[email-masked]>
Cc: NVO3 <[email-masked]>; int-area <[email-masked]>; Service Function Chaining IETF list <[email-masked]>; IETF IPPM WG <[email-masked]>
Subject: Re: [nvo3] [Int-area] [ippm] encapsulation of IOAM data in various protocols - follow up from WG discussion in London

Hi,

"

   [I-D.ietf-nvo3-vxlan-gpe] defines an "O bit" for OAM packets.  Per

   [I-D.ietf-nvo3-vxlan-gpe<https://tools.ietf.org/html/draft-brockners-ippm-ioam-vxlan-gpe-00#ref-I-D.ietf-nvo3-vxlan-gpe>] the O bit indicates that the packet

...FB: What this paragraph states is that IOAM is orthogonal to the O-bit in traffic and will not impact the use of the O-bit. I.e. if the O-bit is set on traffic (because it is OAM traffic), then you’ll continue to have it set with IOAM data added to the packet. If the O-bit isn’t set on the original packet, then it also won’t be set with IOAM data added to the packet.

      header following IOAM protocol.  The value is from the IANA

            registry setup for VXLAN GPE Next Protocol defined in

      [I-D.ietf-nvo3-vxlan-gpe<https://tools.ietf.org/html/draft-brockners-ippm-ioam-vxlan-gpe-00#ref-I-D.ietf-nvo3-vxlan-gpe>]."

...FB: In those cases where IOAM data encapsulation uses the “next protocol” approach, the encapsulation will borrow next-protocol code points from the “parent” protocol. It is the parent protocol that determines how these code points are structured and defined – and in some cases they are 16-bit (like with GRE, where “next protocol” is an Ethertype value) or 8-bit, like for example for “NSH next protocol”

...FB: This is really a deployment question. E.g. if you run VXLAN-GPE over IPv6, you could choose to encapsulate IOAM data natively into IPv6, or you could choose to encapsulate IOAM data into VXLAN-GPE, or even both.

Cheers, Frank

Best regards,

Xiaohu

Regards, Frank

---

From: Mickey Spiegel <[email-masked]>
Date: Fri, 13 Apr 2018 11:22:16 -0700

Tom,

> would be true also for any UDP encapsulation that the network tries to

The intention is to allow for multiple nodes that a packet traverses
to be able to insert IOAM node information in the same trace option,
but leave some flexibility regarding which nodes actually do the
IOAM processing and the node information. This may vary
depending on the transport.

In case of a tunneled encapsulation such as Geneve or VXLAN,
there may still be multiple hops. For example a network may use
Geneve or VXLAN, but only do L2 processing at ToRs, with L3
processing done at aggregation or core switches. In this case
many packets would do 2 Geneve or VXLAN hops, so the packet
would contain IOAM node information from two nodes.

Another example is service function chaining using Geneve or
VXLAN rather than NSH.

> they also have the advantage that the work with any IP protocol or

IPv6 hop-by-hop options has been considered. See
draft-brockners-inband-oam-transport-05. This has not yet been
broken out into a separate draft.

Mickey

> dealing
> One
> >> >> mentioned

---

From: Uma Chunduri <[email-masked]>
Date: Fri, 13 Apr 2018 20:32:33 +0000

[UC]:

        “The
   Segment Routing Header (SRH) for IPv6 offers the ability to transport
   TLV structured data, similar to what NSH does (see
   [I-D.ietf-6man-segment-routing-header]).  In an domain where in-situ
   OAM is used, insertion of the in-situ OAM data is enabled at the
   required edge nodes (i.e. at the in-situ OAM encapsulating/
   decapsulating nodes) by means of configuration.”

I see SRH TLVs are being used for IOAM data. This is fine at the insertion point at the edge node as mentioned above,
but can you help explain how IAOM data gets added along the path towards the egress ? (yes, if you are in hop-by-hop EH, like SRH ).
Only way you can do this is by slapping 8200 encapsulation at every node? Is this correct ?

--
Uma C.

---

From: Tom Herbert <[email-masked]>
Date: Fri, 13 Apr 2018 17:55:34 -0700

Mickey,

Thanks, for the explanation. The requirements around this are not
clear in the drafts so I would suggest that be clarified. Intermediate
nodes should never look at, much less change, transport layer data for
packets for which they are not the addressee (we've already seen
several instances of implemenations skirting these rules out of
convenience!)

One other nit: I don't think a six page I-D really needs twelve
authors! Please consider reducing the number to only the major
contributors.

Tom

>> >> >> discussion topic that we decided to take to the list was the
>> >> >> question
>> >> >> on
>> >> >> encapsulation needs to be as efficient as possible. The “OOAM
>> >> >> header”
>> >> >> is 8
>> >> > meta-data
>> >> > TLV
>> >> >> per

---

From: "Frank Brockners (fbrockne)" <[email-masked]>
Date: Mon, 16 Apr 2018 07:08:16 +0000

Tom,

a quick addition to what Mickey mentioned below: What you seem to have in mind is what draft-ietf-ippm-ioam-data-02 refers to as “layering” (see section 3.), i.e. if you’re running for example Geneve over IPv6, then IOAM data could be encapsulated in both protocols, Geneve and IPv6 – giving you visibility into the “underlay” (IPv6) and the “overlay” (Geneve).

Frank

From: ippm <[email-masked]> On Behalf Of Mickey Spiegel
Sent: Freitag, 13. April 2018 20:22
To: Tom Herbert <[email-masked]>
Cc: NVO3 <[email-masked]>; [email-masked]; Service Function Chaining IETF list <[email-masked]>; IETF IPPM WG <[email-masked]>
Subject: Re: [ippm] [Int-area] encapsulation of IOAM data in various protocols - follow up from WG discussion in London

Tom,

On Thu, Apr 12, 2018 at 10:17 PM, Tom Herbert <[email-masked]<mailto:[email-masked]>> wrote:
Mickey,

Mickey

Thanks,
Tom

On Thu, Apr 12, 2018 at 3:31 PM, Mickey Spiegel
<[email-masked]<mailto:[email-masked]>> wrote:
> Tom,
>
> On Thu, Apr 12, 2018 at 2:46 PM, Tom Herbert <[email-masked]<mailto:[email-masked]>> wrote:
>>
>> On Thu, Apr 12, 2018 at 9:54 AM, Greg Mirsky <[email-masked]<mailto:[email-masked]>>
>> wrote:
>> > Hi Frank,
>> > <[email-masked]<mailto:[email-masked]>> wrote:
>> >>
>> >> whether draft-ooamdt-rtgwg-ooam-header could be leveraged..  After
>> >> carefully

---

From: Tianran Zhou <[email-masked]>
Date: Mon, 16 Apr 2018 08:37:25 +0000

Hi Frank,

How does a forwarder know when and where to insert the data?
In the case of Geneve over IPv6, do you mean the device need to scan all the protocol stack? Or just the outer encapsulation?

Tianran

From: ippm [mailto:[email-masked]] On Behalf Of Frank Brockners (fbrockne)
Sent: Monday, April 16, 2018 3:08 PM
To: Mickey Spiegel <[email-masked]>; Tom Herbert <[email-masked]>
Cc: NVO3 <[email-masked]>; [email-masked]; Service Function Chaining IETF list <[email-masked]>; IETF IPPM WG <[email-masked]>
Subject: Re: [ippm] [Int-area] encapsulation of IOAM data in various protocols - follow up from WG discussion in London

Tom,

Frank

From: ippm <[email-masked]<mailto:[email-masked]>> On Behalf Of Mickey Spiegel
Sent: Freitag, 13. April 2018 20:22
To: Tom Herbert <[email-masked]<mailto:[email-masked]>>
Cc: NVO3 <[email-masked]<mailto:[email-masked]>>; [email-masked]<mailto:[email-masked]>; Service Function Chaining IETF list <[email-masked]<mailto:[email-masked]>>; IETF IPPM WG <[email-masked]<mailto:[email-masked]>>
Subject: Re: [ippm] [Int-area] encapsulation of IOAM data in various protocols - follow up from WG discussion in London

Tom,

Mickey

Thanks,
Tom

---

From: "Frank Brockners (fbrockne)" <[email-masked]>
Date: Mon, 16 Apr 2018 08:51:01 +0000

Hi Tianran,

IOAM is a domain specific feature (see also draft-ietf-ippm-ioam-data-02 sections 3 and 4), which allows an operator to control by means of configuration where and for which traffic IOAM data fields are added/updated/removed from the customer traffic. Using your example of Geneve over IPv6 – with IOAM data in both the Geneve and the IPv6 protocol, one would expect that the operator configures the endpoints of the Geneve tunnel to operate on the IOAM data in Geneve, and the IPv6 routers that the Geneve tunnel traverses to operate on the IOAM data in IPv6.

Frank

From: Tianran Zhou <[email-masked]>
Sent: Montag, 16. April 2018 10:37
To: Frank Brockners (fbrockne) <[email-masked]>; Mickey Spiegel <[email-masked]>; Tom Herbert <[email-masked]>
Cc: NVO3 <[email-masked]>; [email-masked]; Service Function Chaining IETF list <[email-masked]>; IETF IPPM WG <[email-masked]>
Subject: RE: [ippm] [Int-area] encapsulation of IOAM data in various protocols - follow up from WG discussion in London

Hi Frank,

Tianran

To: Mickey Spiegel <[email-masked]<mailto:[email-masked]>>; Tom Herbert <[email-masked]<mailto:[email-masked]>>

Tom,

Frank

Tom,

Mickey

Thanks,
Tom

---

From: Tianran Zhou <[email-masked]>
Date: Mon, 16 Apr 2018 09:06:31 +0000

Hi Frank,

If I recall right, it is not written in the ioam data draft.
Yes, node by node configuration is an easy way. In the draft-zhou-ippm-ioam-yang, we have the “protocol-type” to indicate the layering.
   +--rw ioam
      +--rw ioam-profiles
         +--rw enabled?        boolean
         +--rw ioam-profile* [profile-name]
            +--rw profile-name                    string
            +--rw filter
            |  +--rw filter-type?   ioam-filter-type
            |  +--rw acl-name?      -> /acl:acls/acl/name
            +--rw protocol-type?                  ioam-protocol-type
            +--rw incremental-tracing-profile {incremental-trace}?
            |  ...
            +--rw preallocated-tracing-profile {preallocated-trace}?
            |  ...
            +--rw pot-profile {proof-of-transit}?
            |  ...
            +--rw e2e-profile {edge-to-edge}?
               ...

Tianran
From: Frank Brockners (fbrockne) [mailto:[email-masked]]
Sent: Monday, April 16, 2018 4:51 PM
To: Tianran Zhou <[email-masked]>; Mickey Spiegel <[email-masked]>; Tom Herbert <[email-masked]>
Cc: NVO3 <[email-masked]>; [email-masked]; Service Function Chaining IETF list <[email-masked]>; IETF IPPM WG <[email-masked]>
Subject: RE: [ippm] [Int-area] encapsulation of IOAM data in various protocols - follow up from WG discussion in London

Hi Tianran,

Frank

From: Tianran Zhou <[email-masked]<mailto:[email-masked]>>
Sent: Montag, 16. April 2018 10:37
To: Frank Brockners (fbrockne) <[email-masked]<mailto:[email-masked]>>; Mickey Spiegel <[email-masked]<mailto:[email-masked]>>; Tom Herbert <[email-masked]<mailto:[email-masked]>>
Cc: NVO3 <[email-masked]<mailto:[email-masked]>>; [email-masked]<mailto:[email-masked]>; Service Function Chaining IETF list <[email-masked]<mailto:[email-masked]>>; IETF IPPM WG <[email-masked]<mailto:[email-masked]>>
Subject: RE: [ippm] [Int-area] encapsulation of IOAM data in various protocols - follow up from WG discussion in London

Hi Frank,

Tianran

Tom,

Frank

Tom,

Mickey

Thanks,
Tom

---

From: "Shwetha Bhandari (shwethab)" <[email-masked]>
Date: Mon, 16 Apr 2018 10:16:44 +0000

Hi Tianran,

> If I recall right, it is not written in the ioam data draft.

Data draft is defining the data to be carried in IOAM in an encapsulation agnostic way, it does not specify how the encapsulation protocol is configured.

> Yes, node by node configuration is an easy way.

While it is, it does not have to be a node by node configuration. It can be part of the encapsulation definition. 

For e.g. If the encapsulation is IPv6 and if we define the data to be carried as HbH options, then based on the Option Type with highest order 2 bits set to 00 then the v6 nodes that implement IOAM will process the option and others will skip over.

Thanks,

Shwetha

From: ippm <[email-masked]> on behalf of Tianran Zhou <[email-masked]>
Date: Monday, April 16, 2018 at 2:36 PM
To: "Frank Brockners (fbrockne)" <[email-masked]>, Mickey Spiegel <[email-masked]>, Tom Herbert <[email-masked]>
Cc: NVO3 <[email-masked]>, "[email-masked]" <[email-masked]>, Service Function Chaining IETF list <[email-masked]>, IETF IPPM WG <[email-masked]>
Subject: Re: [ippm] [Int-area] encapsulation of IOAM data in various protocols - follow up from WG discussion in London

Hi Frank,

   +--rw ioam

      +--rw ioam-profiles

         +--rw enabled?        boolean

            +--rw filter

            |  ...

            |  ...

            |  ...

               ...

Tianran

Hi Tianran,

Frank

Hi Frank,

Tianran

 

Tom,

Frank

Tom,

Mickey,

VXLAN rather than NSH.

Mickey

Thanks,
Tom

---

From: Tianran Zhou <[email-masked]>
Date: Mon, 16 Apr 2018 13:31:55 +0000

Hi Shwetha,

You are talking about the outer encapsution. It is straight forward for the underlay to record by the header. But what about the overlay, i.e., inner encapsulation(e.g. geneve)? Without special configuration, intermediate node will not read the inner header, hence not be able to process IOAM.
Maybe we are not synced by this overlay/underlay use case. :-)

Tianran

________________________________
Sent from WeLink

发件人： Shwetha Bhandari (shwethab)
收件人： Tianran Zhou<[email-masked]<mailto:[email-masked]>>;Frank Brockners (fbrockne)<[email-masked]<mailto:[email-masked]>>;Mickey Spiegel<[email-masked]<mailto:[email-masked]>>;Tom Herbert<[email-masked]<mailto:[email-masked]>>
抄送： NVO3<[email-masked]<mailto:[email-masked]>>;int-area<[email-masked]<mailto:[email-masked]>>;Service Function Chaining IETF list<[email-masked]<mailto:[email-masked]>>;IETF IPPM WG<[email-masked]<mailto:[email-masked]>>
主题： Re: [ippm] [Int-area] encapsulation of IOAM data in various protocols - follow up from WG discussion in London
时间： 2018-04-16 18:17:01

Hi Tianran,

Thanks,
Shwetha

Hi Frank,

Tianran

Hi Tianran,

Frank

Hi Frank,

Tianran

Tom,

Frank

Tom,

Mickey

Thanks,
Tom

---

From: Tom Herbert <[email-masked]>
Date: Mon, 16 Apr 2018 07:40:20 -0700

On Mon, Apr 16, 2018 at 6:31 AM, Tianran Zhou <[email-masked]> wrote:
> Hi Shwetha,
>
> You are talking about the outer encapsution. It is straight forward for the
> underlay to record by the header. But what about the overlay, i.e., inner
> encapsulation(e.g. geneve)? Without special configuration, intermediate node
> will not read the inner header, hence not be able to process IOAM.e

Hi Tianran,

I believe that is also not protocol conformant. Intermediate nodes
should not be processing transport layer data as this can lead to
misinterpretation and possibly silent data corruption.

For instance, Geneve is a UDP encapsulation protocol with assigned
port 6081. In order for an intermediate device to process the Geneve
encapsulation header it would need to look for packets with
destination port of 6081 since that is the only possible
discriminator. However, transport port numbers do not have global
meaning and hosts may use port numbers for other purposes (RFC7605
describes this). So a packet to port 6081 might be something other
than Geneve and may be misinterpreted. If a misinterpreted packet is
changed (like ippm data is written) then that would be systematic
silent data corruption.

As far as I know, hop-by-hop options is the only protocol confirming
mechanism that allows an intermediate note to change data of packet in
flight. Encpasulation is the only conforming mechanism that allows an
intermediate node to add data (like extension headers) to a packet in
flight.

Tom

> 收件人： Tianran Zhou<[email-masked]>;Frank Brockners
> (fbrockne)<[email-masked]>;Mickey
> Spiegel<[email-masked]>;Tom Herbert<[email-masked]>
> 抄送： NVO3<[email-masked]>;int-area<[email-masked]>;Service Function
> Chaining IETF list<[email-masked]>;IETF IPPM WG<[email-masked]>
> Data draft is defining the data to be carried in IOAM in an encapsulation
> agnostic way, it does not specify how the encapsulation protocol is
> configured.
>
>
>
> For e.g. If the encapsulation is IPv6 and if we define the data to be
> carried as HbH options, then based on the Option Type with highest order 2
> bits set to 00 then the v6 nodes that implement IOAM will process the option
> and others will skip over.
>
>
>
>
>
> Thanks,
>
> Shwetha
>
>
>
> To: "Frank Brockners (fbrockne)" <[email-masked]>, Mickey Spiegel
> <[email-masked]>, Tom Herbert <[email-masked]>
> Cc: NVO3 <[email-masked]>, "[email-masked]" <[email-masked]>, Service
> Function Chaining IETF list <[email-masked]>, IETF IPPM WG <[email-masked]>
> Subject: Re: [ippm] [Int-area] encapsulation of IOAM data in various
> protocols - follow up from WG discussion in London
>
>
>
> Hi Frank,
>
>
>
> Yes, node by node configuration is an easy way. In the
> draft-zhou-ippm-ioam-yang, we have the “protocol-type” to indicate the
> layering.
>
>    +--rw ioam
>
>       +--rw ioam-profiles
>
>          +--rw enabled?        boolean
>
> To: Tianran Zhou <[email-masked]>; Mickey Spiegel
> <[email-masked]>; Tom Herbert <[email-masked]>
> Cc: NVO3 <[email-masked]>; [email-masked]; Service Function Chaining IETF
> list <[email-masked]>; IETF IPPM WG <[email-masked]>
> Subject: RE: [ippm] [Int-area] encapsulation of IOAM data in various
> protocols - follow up from WG discussion in London
>
>
>
> Hi Tianran,
>
>
>
> IOAM is a domain specific feature (see also draft-ietf-ippm-ioam-data-02
> sections 3 and 4), which allows an operator to control by means of
> configuration where and for which traffic IOAM data fields are
> added/updated/removed from the customer traffic. Using your example of
> Geneve over IPv6 – with IOAM data in both the Geneve and the IPv6 protocol,
> one would expect that the operator configures the endpoints of the Geneve
> tunnel to operate on the IOAM data in Geneve, and the IPv6 routers that the
> Geneve tunnel traverses to operate on the IOAM data in IPv6.
>
>
>
> Frank
>
>
>
> To: Frank Brockners (fbrockne) <[email-masked]>; Mickey Spiegel
> <[email-masked]>; Tom Herbert <[email-masked]>
> Cc: NVO3 <[email-masked]>; [email-masked]; Service Function Chaining IETF
> list <[email-masked]>; IETF IPPM WG <[email-masked]>
> Subject: RE: [ippm] [Int-area] encapsulation of IOAM data in various
> protocols - follow up from WG discussion in London
>
>
>
> Hi Frank,
>
>
>
> In the case of Geneve over IPv6, do you mean the device need to scan all the
> protocol stack? Or just the outer encapsulation?
>
>
>
> Tianran
>
>
>
> Cc: NVO3 <[email-masked]>; [email-masked]; Service Function Chaining IETF
> list <[email-masked]>; IETF IPPM WG <[email-masked]>
> Subject: Re: [ippm] [Int-area] encapsulation of IOAM data in various
> protocols - follow up from WG discussion in London
>
>
>
>
>
> Tom,
>
>
>
> a quick addition to what Mickey mentioned below: What you seem to have in
> mind is what draft-ietf-ippm-ioam-data-02 refers to as “layering” (see
> section 3.), i.e. if you’re running for example Geneve over IPv6, then IOAM
> data could be encapsulated in both protocols, Geneve and IPv6 – giving you
> visibility into the “underlay” (IPv6) and the “overlay” (Geneve).
>
>
>
> Frank
>
>
>
> Cc: NVO3 <[email-masked]>; [email-masked]; Service Function Chaining IETF
> list <[email-masked]>; IETF IPPM WG <[email-masked]>
> Subject: Re: [ippm] [Int-area] encapsulation of IOAM data in various
> protocols - follow up from WG discussion in London
>
>
>
> Tom,
>
>
>

---

From: Tianran Zhou <[email-masked]>
Date: Tue, 17 Apr 2018 01:17:59 +0000

I think it's better that Frank or Shwetha can explain the multi-layer use case in detail.

Tianran
> From: Tom Herbert [mailto:[email-masked]]
> Sent: Monday, April 16, 2018 10:40 PM
> To: Tianran Zhou <[email-masked]>
> Cc: Shwetha Bhandari (shwethab) <[email-masked]>; Frank Brockners
> (fbrockne) <[email-masked]>; Mickey Spiegel
> <[email-masked]>; NVO3 <[email-masked]>; int-area
> <[email-masked]>; Service Function Chaining IETF list <[email-masked]>;
> IETF IPPM WG <[email-masked]>
> > You are talking about the outer encapsution. It is straight forward
> > for the underlay to record by the header. But what about the overlay,
> > i.e., inner encapsulation(e.g. geneve)? Without special configuration,
> > intermediate node will not read the inner header, hence not be able to
> > process IOAM.e
> 
> Hi Tianran,
> 
> I believe that is also not protocol conformant. Intermediate nodes should
> not be processing transport layer data as this can lead to misinterpretation
> and possibly silent data corruption.
> 
> For instance, Geneve is a UDP encapsulation protocol with assigned port 6081.
> In order for an intermediate device to process the Geneve encapsulation header
> it would need to look for packets with destination port of 6081 since that
> is the only possible discriminator. However, transport port numbers do not
> have global meaning and hosts may use port numbers for other purposes (RFC7605
> describes this). So a packet to port 6081 might be something other than Geneve
> and may be misinterpreted. If a misinterpreted packet is changed (like ippm
> data is written) then that would be systematic silent data corruption.
> 
> As far as I know, hop-by-hop options is the only protocol confirming mechanism
> that allows an intermediate note to change data of packet in flight.
> Encpasulation is the only conforming mechanism that allows an intermediate
> node to add data (like extension headers) to a packet in flight.
> 
> Tom
> 
> > 主题： Re: [ippm] [Int-area] encapsulation of IOAM data in various
> > Data draft is defining the data to be carried in IOAM in an
> > encapsulation agnostic way, it does not specify how the encapsulation
> > protocol is configured.
> >
> >
> >
> > While it is, it does not have to be a node by node configuration. It
> > can be part of the encapsulation definition.
> >
> > carried as HbH options, then based on the Option Type with highest
> > order 2 bits set to 00 then the v6 nodes that implement IOAM will
> > process the option and others will skip over.
> >
> >
> >
> >
> >
> > Thanks,
> >
> > Shwetha
> >
> >
> >
> > Cc: NVO3 <[email-masked]>, "[email-masked]" <[email-masked]>,
> > Service Function Chaining IETF list <[email-masked]>, IETF IPPM WG
> > <[email-masked]>
> > Cc: NVO3 <[email-masked]>; [email-masked]; Service Function Chaining
> > IETF list <[email-masked]>; IETF IPPM WG <[email-masked]>
> > IOAM is a domain specific feature (see also
> > draft-ietf-ippm-ioam-data-02 sections 3 and 4), which allows an
> > operator to control by means of configuration where and for which
> > traffic IOAM data fields are added/updated/removed from the customer
> > traffic. Using your example of Geneve over IPv6 – with IOAM data in
> > both the Geneve and the IPv6 protocol, one would expect that the
> > operator configures the endpoints of the Geneve tunnel to operate on
> > the IOAM data in Geneve, and the IPv6 routers that the Geneve tunnel
> traverses to operate on the IOAM data in IPv6.
> >
> >
> >
> > Frank
> >
> >
> >
> > Cc: NVO3 <[email-masked]>; [email-masked]; Service Function Chaining
> > IETF list <[email-masked]>; IETF IPPM WG <[email-masked]>
> > In the case of Geneve over IPv6, do you mean the device need to scan
> > all the protocol stack? Or just the outer encapsulation?
> >
> >
> >
> > Tianran
> >
> >
> >
> > Cc: NVO3 <[email-masked]>; [email-masked]; Service Function Chaining
> > IETF list <[email-masked]>; IETF IPPM WG <[email-masked]>
> > a quick addition to what Mickey mentioned below: What you seem to have
> > in mind is what draft-ietf-ippm-ioam-data-02 refers to as “layering”
> > (see section 3.), i.e. if you’re running for example Geneve over IPv6,
> > then IOAM data could be encapsulated in both protocols, Geneve and
> > IPv6 – giving you visibility into the “underlay” (IPv6) and the “overlay”
> (Geneve).
> >
> >
> >
> > Frank
> >
> >
> >
> > Cc: NVO3 <[email-masked]>; [email-masked]; Service Function Chaining
> > IETF list <[email-masked]>; IETF IPPM WG <[email-masked]>
> >>> >> draft-brockners-ippm-ioam-geneve-00,
> >>> >> draft-weis-ippm-ioam-gre-00). One discussion topic that we
> >>> >> decided to take to the list was the question on whether
> >>> >> draft-ooamdt-rtgwg-ooam-header could be leveraged..  After
> >>> >> carefully considering draft-ooamdt-rtgwg-ooam-header, I came to
> >>> >> the conclusion that the “OOAM header” does not meet the needs of
> >>> >> IOAM:
> >>> >>
> >>> I'm missing something here. I looked at the drafts you referenced
> >>> and each of them looks like the overhead for OAM is greater that
> >>> four bytes. In each there is some overhead equivalent to
> >>> type/length, for instance in Geneve four bytes are needed for option
> >>> class, type, and length. Unless the the OAM data is zero length, I
> >>> don't see how this adds up to only four bytes of overhead.
> >>
> >>
> >> The four versus eight bytes just refers to the fields in the four
> >> bytes of IOAM info, that is common to all IOAM options. Beyond that,
> >> there are IOAM option specific fields. For example if doing one of
> >> the IOAM trace options, there are four bytes of trace option header,
> >> including the IOAM-trace-type, NodeLen, Flags, and RemainingLen
> >> fields. These are followed by the node data list containing the per
> >> hop IOAM information.
> >>
> >> In looking at the OOAM header content, it has nothing to do with any
> >> of the IOAM information after the first four bytes. It contains
> >> another variant of the information in the first four bytes of IOAM
> >> info, spread out over eight bytes.
> >>
> >>>
> >>> Tom
> >>>
> >>> >
> >> The flags field only has one defined flag at the moment, for a
> >> timestamp block. For IOAM trace we need per hop timestamps, which the
> >> timestamp block cannot address, i.e. the timestamp block is redundant for
> IOAM.
> >>
> >>>
> >> IMO, taking the first two bits of the IOAM-Type to define a Version
> >> field would be a good thing. This does not require adding four more
> >> bytes of overhead. 64 IOAM-Types is more than enough.
> >>
> >>>
> >>> >>
> >>> > shim, I don't see problem in making them use OOAM Header. Would
> >>> > you agree?
> >>> >
> >>> >>
> >>> >> into use 16-bits for their next protocol code points. See e.g.
> >>> >> the GRE encapsulation – as specified in
> >>> >> With the above in mind, I’d suggest that the WG moves forward
> >>> >> with specific definitions for encapsulating IOAM data into
> >>> >> protocols – per the above mentioned drafts.
> >>> >>
> >>> >>
> >>> >>
> >>> >> Regards, Frank
> >>> >>
> >>> >>

---

From: "Frank Brockners (fbrockne)" <[email-masked]>
Date: Tue, 17 Apr 2018 07:51:57 +0000

Hi Tianran,

Tom's note already includes the hint: You'll add IOAM data to the protocol/layer that you're interested in monitoring. Again using Geneve over IPv6 as an example: 
* If you're interested in the overlay, i.e. Geneve (e.g. timestamping the packet when it enters and exists the tunnel) - you'd add IOAM data to Geneve
* If you're interested in the underlay, i.e. IPv6 (e.g. you'd like to understand which path packets take in the v6 network) - you'd add IOAM data to IPv6
* If you're interested in both, then you'd add IOAM data to Geneve and IPv6

Draft draft-ietf-ippm-ioam-data-02 already mentions layering (see section 3):
"Layering: If several encapsulation protocols (e.g., in case of tunneling) are stacked on top of each other, IOAM data-records could be present at every layer.  The behavior follows the ships-in-the-night model."

Given the discussion here, we'll add some additional text in the next revision to make things crisper (e.g. adding an example might help).

Frank

-----Original Message-----
From: Tianran Zhou <[email-masked]> 
Sent: Dienstag, 17. April 2018 03:18
To: Tom Herbert <[email-masked]>
Cc: Shwetha Bhandari (shwethab) <[email-masked]>; Frank Brockners (fbrockne) <[email-masked]>; Mickey Spiegel <[email-masked]>; NVO3 <[email-masked]>; Service Function Chaining IETF list <[email-masked]>; IETF IPPM WG <[email-masked]>
Subject: RE: [ippm] [Int-area] encapsulation of IOAM data in various protocols - follow up from WG discussion in London

Tianran
> <[email-masked]>; Service Function Chaining IETF list 
> <[email-masked]>; IETF IPPM WG <[email-masked]>
> > for the underlay to record by the header. But what about the 
> > overlay, i.e., inner encapsulation(e.g. geneve)? Without special 
> > configuration, intermediate node will not read the inner header, 
> > hence not be able to process IOAM.e
> 
> Hi Tianran,
> 
> In order for an intermediate device to process the Geneve 
> than Geneve and may be misinterpreted. If a misinterpreted packet is changed (like ippm data is written) then that would be systematic silent data corruption.
> 
> mechanism that allows an intermediate note to change data of packet in flight.
> Encpasulation is the only conforming mechanism that allows an 
> intermediate node to add data (like extension headers) to a packet in flight.
> 
> Tom
> 
> > encapsulation agnostic way, it does not specify how the 
> > encapsulation protocol is configured.
> >
> >
> >
> > For e.g. If the encapsulation is IPv6 and if we define the data to 
> > be carried as HbH options, then based on the Option Type with 
> > highest order 2 bits set to 00 then the v6 nodes that implement IOAM 
> > will process the option and others will skip over.
> >
> >
> >
> >
> >
> > Thanks,
> >
> > Shwetha
> >
> >
> >
> > To: "Frank Brockners (fbrockne)" <[email-masked]>, Mickey 
> > Spiegel <[email-masked]>, Tom Herbert 
> > <[email-masked]>
> > draft-zhou-ippm-ioam-yang, we have the “protocol-type” to indicate 
> > the layering.
> >
> >    +--rw ioam
> >
> >       +--rw ioam-profiles
> >
> >          +--rw enabled?        boolean
> >
> > Cc: NVO3 <[email-masked]>; [email-masked]; Service Function 
> > Chaining IETF list <[email-masked]>; IETF IPPM WG <[email-masked]>
> > Cc: NVO3 <[email-masked]>; [email-masked]; Service Function 
> > Chaining IETF list <[email-masked]>; IETF IPPM WG <[email-masked]>
> > From: ippm [mailto:[email-masked]] On Behalf Of Frank 
> > Brockners
> > (fbrockne)
> > Cc: NVO3 <[email-masked]>; [email-masked]; Service Function 
> > Chaining IETF list <[email-masked]>; IETF IPPM WG <[email-masked]>
> > a quick addition to what Mickey mentioned below: What you seem to 
> > have in mind is what draft-ietf-ippm-ioam-data-02 refers to as “layering”
> > (see section 3.), i.e. if you’re running for example Geneve over 
> > IPv6, then IOAM data could be encapsulated in both protocols, Geneve 
> > and
> > Cc: NVO3 <[email-masked]>; [email-masked]; Service Function 
> > Chaining IETF list <[email-masked]>; IETF IPPM WG <[email-masked]>
> > In draft-brockners-ippm-ioam-geneve-00 for instance, there is the 
> > text in the introduction:
> >
> > I assume this means that as packets with Geneve encapsulation 
> > traverse the network they are interpreted by intermediate nodes as 
> > being Geneve. Since Geneve is a UDP encapsulation, then the 
> > destination UDP port number would be used to identify packets as 
> > being Geneve. So an intermediate device might be looking for UDP 
> > packets destined to port
> > protocol that would allow this seems pretty incorrect! Note that 
> > this would be true also for any UDP encapsulation that the network 
> > tries to interpret.
> >
> >
> >
> >>> On Thu, Apr 12, 2018 at 9:54 AM, Greg Mirsky 
> >>> <[email-masked]>
> >>> wrote:
> >>> > Hi Frank,
> >>> > GIM>>. I believe that this is very much relevant to work of 
> >>> > GIM>>other
> >>> > working
> >>> >> dealing with the encapsulation of IOAM data in various 
> >>> >> protocols (draft-brockners-ippm-ioam-vxlan-gpe-00,
> >>> >> bytes long. The approach for IOAM data encapsulation in the 
> >>> >> above mentioned drafts only requires 4 bytes. Using the OOAM 
> >>> >> header approach would add an unnecessary overhead of 4 bytes – 
> >>> >> which is significant.
> >>> Greg,
> >>>
> >>> type/length, for instance in Geneve four bytes are needed for 
> >>> option class, type, and length. Unless the the OAM data is zero 
> >>> length, I don't see how this adds up to only four bytes of overhead.
> >>
> >>
> >> bytes of IOAM info, that is common to all IOAM options. Beyond 
> >> that, there are IOAM option specific fields. For example if doing 
> >> one of the IOAM trace options, there are four bytes of trace option 
> >> header, including the IOAM-trace-type, NodeLen, Flags, and 
> >> RemainingLen fields. These are followed by the node data list 
> >> In looking at the OOAM header content, it has nothing to do with 
> >> any of the IOAM information after the first four bytes. It contains 
> >> timestamp block. For IOAM trace we need per hop timestamps, which 
> >> the timestamp block cannot address, i.e. the timestamp block is 
> >> redundant for
> IOAM.
> >>
> >>>
> >>> >> shown at recent IETF hackathons – and we’re expecting 
> >>> >> additional implementations to be publicized soon. Interoperable 
> >>> >> implementations need timely specifications. Despite the 
> >>> >> question being asked, the recent thread on OOAM in the NVO3 
> >>> >> list hasn’t revealed any implementation of the OOAM header.
> >>> >> In
> >>> >> addition, the thread revealed that several fundamental 
> >>> >> questions about the OOAM header are still open, such as whether 
> >>> >> or how active OAM mechanisms within protocols such as Geneve 
> >>> >> would apply to the OOAM header. This ultimately means that we 
> >>> >> won’t get to a timely specification.
> >>> >
> >>> >> protocol. Some protocols that IOAM data needs to be 
> >>> >> encapsulated into use 16-bits for their next protocol code points. See e.g.
> >>> >    [I-D.ietf-nvo3-geneve], BIER 
> >>> > [I-D.ietf-bier-mpls-encapsulation],
> and
> >>> > statemnt are quite clear in identifying the scope of networks 
> >>> > that may benefit from using not only common OOAM Header but 
> >>> > common OOAM mechanisms, e.g. Echo Request/Reply.
> >>> >

---

From: Tom Herbert <[email-masked]>
Date: Tue, 17 Apr 2018 07:22:45 -0700

> * If you're interested in both, then you'd add IOAM data to Geneve and IPv6
Frank,

In that case why not just use a hop-by-hop option for measuring the
underlay and a destination option for measuring the overlay? The
advantage is that this works _any_ IP encapsulation method or any IP
protocol for that matter. I don't believe adding ippm to every
encapsulation protocol is straightforward: e.g.
draft-brockners-ippm-ioam-geneve describe but notes the limited size
of header, draft-weis-ippm-ioam-gre states that a new EtherType would
be needed just for this purpose. This also entails additional
encapsulation-specific HW support also, whereas support destination
and hbh options could be more generic.

Tom

---

From: "Frank Brockners (fbrockne)" <[email-masked]>
Date: Wed, 18 Apr 2018 07:51:40 +0000

Tom,

inline... ("...FB")

-----Original Message-----
From: Tom Herbert <[email-masked]> 
Sent: Dienstag, 17. April 2018 16:23
To: Frank Brockners (fbrockne) <[email-masked]>
Cc: Tianran Zhou <[email-masked]>; Shwetha Bhandari (shwethab) <[email-masked]>; Mickey Spiegel <[email-masked]>; NVO3 <[email-masked]>; Service Function Chaining IETF list <[email-masked]>; IETF IPPM WG <[email-masked]>
Subject: Re: [ippm] [Int-area] encapsulation of IOAM data in various protocols - follow up from WG discussion in London

On Tue, Apr 17, 2018 at 12:51 AM, Frank Brockners (fbrockne) <[email-masked]> wrote:
> * If you're interested in the overlay, i.e. Geneve (e.g. timestamping 
> the packet when it enters and exists the tunnel) - you'd add IOAM data 
> to Geneve
> * If you're interested in the underlay, i.e. IPv6 (e.g. you'd like to 
> understand which path packets take in the v6 network) - you'd add IOAM 
> data to IPv6
Frank,

In that case why not just use a hop-by-hop option for measuring the underlay and a destination option for measuring the overlay? The advantage is that this works _any_ IP encapsulation method or any IP protocol for that matter. I don't believe adding ippm to every encapsulation protocol is straightforward: e.g.
draft-brockners-ippm-ioam-geneve describe but notes the limited size of header, draft-weis-ippm-ioam-gre states that a new EtherType would be needed just for this purpose. This also entails additional encapsulation-specific HW support also, whereas support destination and hbh options could be more generic.

...FB: There are quite a few deployment examples, such as overlay VPN services, where you don't have access to the underlay (e.g. IPv6) - but do control the overlay and desire insights into your overlay using IOAM. Hence the need carry IOAM data along the overlay encapsulation. 

Frank 

Tom

> <[email-masked]>; NVO3 <[email-masked]>; Service 
> Function Chaining IETF list <[email-masked]>; IETF IPPM WG 
> <[email-masked]>
>> > 抄送： NVO3<[email-masked]>;int-area<[email-masked]>;Service 
>> > Function Chaining IETF list<[email-masked]>;IETF IPPM 
>> > WG<[email-masked]>
>> > While it is, it does not have to be a node by node configuration. 
>> > It can be part of the encapsulation definition.
>> >
>> > highest order 2 bits set to 00 then the v6 nodes that implement 
>> > IOAM will process the option and others will skip over.
>> >
>> >
>> >
>> >
>> >
>> > Thanks,
>> >
>> > Shwetha
>> >
>> >
>> >
>> > Subject: Re: [ippm] [Int-area] encapsulation of IOAM data in 
>> > various protocols - follow up from WG discussion in London
>> >
>> >
>> >
>> > Hi Frank,
>> >
>> >
>> >
>> > Subject: RE: [ippm] [Int-area] encapsulation of IOAM data in 
>> > various protocols - follow up from WG discussion in London
>> >
>> >
>> >
>> > Hi Tianran,
>> >
>> >
>> >
>> > traffic IOAM data fields are added/updated/removed from the 
>> > customer traffic. Using your example of Geneve over IPv6 – with 
>> > IOAM data in both the Geneve and the IPv6 protocol, one would 
>> > expect that the operator configures the endpoints of the Geneve 
>> > tunnel to operate on the IOAM data in Geneve, and the IPv6 routers 
>> > that the Geneve tunnel
>> > Subject: RE: [ippm] [Int-area] encapsulation of IOAM data in 
>> > various protocols - follow up from WG discussion in London
>> >
>> >
>> >
>> > Hi Frank,
>> >
>> >
>> >
>> > In the case of Geneve over IPv6, do you mean the device need to 
>> > scan all the protocol stack? Or just the outer encapsulation?
>> >
>> >
>> >
>> > Tianran
>> >
>> >
>> >
>> > Subject: Re: [ippm] [Int-area] encapsulation of IOAM data in 
>> > various protocols - follow up from WG discussion in London
>> >
>> >
>> >
>> >
>> >
>> > Tom,
>> >
>> >
>> >
>> > IPv6, then IOAM data could be encapsulated in both protocols, 
>> > Geneve and
>> > Subject: Re: [ippm] [Int-area] encapsulation of IOAM data in 
>> > various protocols - follow up from WG discussion in London
>> >
>> >
>> >
>> > Tom,
>> >
>> >
>> >
>> > -- except at the endpoints -- may be incorrect, because port 
>> > numbers are meaningful only at the endpoints."
>> >
>> > misinterpretation may also mean silent data corruption of packets. 
>> > A protocol that would allow this seems pretty incorrect! Note that 
>> >>> >> Back at the IPPM meeting in London, we discussed several 
>> >>> >> drafts dealing with the encapsulation of IOAM data in various 
>> >>> >> carefully considering draft-ooamdt-rtgwg-ooam-header, I came 
>> >>> >> to the conclusion that the “OOAM header” does not meet the 
>> >>> >> needs of
>> >>> >> IOAM:
>> >>> >>
>> >> one of the IOAM trace options, there are four bytes of trace 
>> >> option header, including the IOAM-trace-type, NodeLen, Flags, and 
>> >> any of the IOAM information after the first four bytes. It 
>> >> contains another variant of the information in the first four 
>> >> bytes of IOAM info, spread out over eight bytes.
>> >>
>> >>>
>> >>> Tom
>> >>>
>> >>> >
>> >> IMO, taking the first two bits of the IOAM-Type to define a 
>> >> Version field would be a good thing. This does not require adding 
>> >> four more bytes of overhead. 64 IOAM-Types is more than enough.
>> >>
>> >>>
>> >>> >>
>> >>> >> additional implementations to be publicized soon. 
>> >>> >> Interoperable implementations need timely specifications. 
>> >>> >> Despite the question being asked, the recent thread on OOAM in 
>> >>> >> the NVO3 list hasn’t revealed any implementation of the OOAM header.
>> >>> >> In
>> >>> >> questions about the OOAM header are still open, such as 
>> >>> >> whether or how active OAM mechanisms within protocols such as 
>> >>> >> Geneve would apply to the OOAM header. This ultimately means 
>> >>> >> that we won’t get to a timely specification.
>> >>> >
>> >>> > these implementations already updated to the newly proposed 
>> >>> > iOAM shim, I don't see problem in making them use OOAM Header. 
>> >>> > Would you agree?
>> >>> >
>> >>> >>

---

From: Tom Herbert <[email-masked]>
Date: Thu, 19 Apr 2018 07:32:02 -0700

Frank,

I'm not sure I follow your argument. Maybe examining some of the
scenarios would help (here in in the draft):

Consider that an end host sources a simple TCP packet. So headers look like

  IP | TCP

If they wish to do ippm for end-to-end measure, a destination option
can be used, so packet looks like

  IP | ippm_DO | TCP

Similarly, if they want to do per hop measurement they that could also
use and appropiate HBH option

  IP | ippm_HBH | ippm_DO | TCP

Now consider that a device in the network is tunneling the packet with
encapsulation.

Basic encapsulated packet looks like

  IP | encap_hdr | IP | TCP

Measurements across the tunnel (ingress to egress point) could be done

  IP | ippm_DO | encap_hdr | IP | TCP (#1)

Measuring underday then would be

  IP | ippm_HBH | encap_hdr | IP | TCP

And again both can be done simulataneously

  IP | ippm_HBH | ippm_DO | encap_hdr | IP | TCP

Note also that in simple ipip encasulation the encap_hdr would be null
when the DO EH hae next protocol as IP. So that case would look like:

  IP | ippm_HBH | ippm_DO | IP | TCP

Now if the ippm is in the encapsulation, then to measure across the
tunnel packet (ingress to egress) looks something like

   IP | encap_hdr | ippm_data | IP | TCP (#2)

Where ippm_data is either a field or TLV of the encapsulation (like
would be in Geneve) or it's a layer in a header chain (like in the GRE
proposal).

If underlay is also measured also then that gives

  IP | ippm_HBH | encap_hdr | ippm_data | IP | TCP
'
#1 and #2 above are the comparable cases.

#2 (ippm in the encapsulation) measures from encapsulation to
decapsulation of the tunnel. This corresponds to the tunnel ingress
and egress points. In #1, the DO is used to measure between tunnel
ingress and egress. Processing of the encapsulation header is also
done at tunnel ingress and egress so that coincides with where DO is
processed. In an implementation, the destination option can even be
passed the encpasulation layer if desired so that it can be processed
as though it were part of the encapsulation.

So from this, I don't see any functional difference between using the
ippm DO in the outer IP header of an encapsulation and putting the
ippm data in the encapsulation. There might be an agument that this is
needed to support IPv4, but even so I'm not sure that justifies
retrofitting every IETF defined encapsulation protocol to support
ippm.

Also, I think how ippm works with simple IPIP encapsulation should be
thought out since that is still probably the most common encapsulation
deployed. There is no way to extend IPIP (doesn't even have encap
header) and adding a new IP protocol code point (similar to
draft-weis-ippm-ioam-gre where a new EtherType for this purpose is
proposed) is pretty unlikely to be accepted. So in this case of IPIP,
it seems like the only answer is to use DO or HBH options, but as I
mentioned I don't see that is any disadvantage or less functional of a
solution (except for lacking IPv4 support).

Tom

---

From: "Frank Brockners (fbrockne)" <[email-masked]>
Date: Thu, 19 Apr 2018 14:47:10 +0000

Tom,

-----Original Message-----
From: Tom Herbert <[email-masked]> 
Sent: Donnerstag, 19. April 2018 16:32
To: Frank Brockners (fbrockne) <[email-masked]>
Cc: Tianran Zhou <[email-masked]>; Shwetha Bhandari (shwethab) <[email-masked]>; Mickey Spiegel <[email-masked]>; NVO3 <[email-masked]>; Service Function Chaining IETF list <[email-masked]>; IETF IPPM WG <[email-masked]>
Subject: Re: [ippm] [Int-area] encapsulation of IOAM data in various protocols - follow up from WG discussion in London

On Wed, Apr 18, 2018 at 12:51 AM, Frank Brockners (fbrockne) <[email-masked]> wrote:
> Cc: Tianran Zhou <[email-masked]>; Shwetha Bhandari (shwethab) 
> <[email-masked]>; Mickey Spiegel <[email-masked]>; 
> NVO3 <[email-masked]>; Service Function Chaining IETF list 
>> the packet when it enters and exists the tunnel) - you'd add IOAM 
>> data to Geneve
>> understand which path packets take in the v6 network) - you'd add 
>> IOAM data to IPv6
>> * If you're interested in both, then you'd add IOAM data to Geneve 
>> and
>> IPv6
>>
> Frank,
>
Frank,

I'm not sure I follow your argument. Maybe examining some of the scenarios would help (here in in the draft):

  IP | TCP

If they wish to do ippm for end-to-end measure, a destination option can be used, so packet looks like

  IP | ippm_DO | TCP

Similarly, if they want to do per hop measurement they that could also use and appropiate HBH option

Now consider that a device in the network is tunneling the packet with encapsulation.

Note also that in simple ipip encasulation the encap_hdr would be null when the DO EH hae next protocol as IP. So that case would look like:

Now if the ippm is in the encapsulation, then to measure across the tunnel packet (ingress to egress) looks something like

Where ippm_data is either a field or TLV of the encapsulation (like would be in Geneve) or it's a layer in a header chain (like in the GRE proposal).

  IP | ippm_HBH | encap_hdr | ippm_data | IP | TCP '
#1 and #2 above are the comparable cases.

#2 (ippm in the encapsulation) measures from encapsulation to decapsulation of the tunnel. This corresponds to the tunnel ingress and egress points. In #1, the DO is used to measure between tunnel ingress and egress. Processing of the encapsulation header is also done at tunnel ingress and egress so that coincides with where DO is processed. In an implementation, the destination option can even be passed the encpasulation layer if desired so that it can be processed as though it were part of the encapsulation.

So from this, I don't see any functional difference between using the ippm DO in the outer IP header of an encapsulation and putting the ippm data in the encapsulation. There might be an agument that this is needed to support IPv4, but even so I'm not sure that justifies retrofitting every IETF defined encapsulation protocol to support ippm.

Also, I think how ippm works with simple IPIP encapsulation should be thought out since that is still probably the most common encapsulation deployed. There is no way to extend IPIP (doesn't even have encap
header) and adding a new IP protocol code point (similar to draft-weis-ippm-ioam-gre where a new EtherType for this purpose is
proposed) is pretty unlikely to be accepted. So in this case of IPIP, it seems like the only answer is to use DO or HBH options, but as I mentioned I don't see that is any disadvantage or less functional of a solution (except for lacking IPv4 support).

...FB: Thanks. I do follow your logic, though ignoring v4 isn't really an option - hence the need to get IOAM data into the tunnel encaps - at least into the most popular ones. 

Frank

Tom

>>> > You are talking about the outer encapsution. It is straight 
>>> > forward for the underlay to record by the header. But what about 
>>> > the overlay, i.e., inner encapsulation(e.g. geneve)? Without 
>>> > special configuration, intermediate node will not read the inner 
>>> > header, hence not be able to process IOAM.e
>>>
>>> Hi Tianran,
>>>
>>> > To: Frank Brockners (fbrockne) <[email-masked]>; Mickey 
>>> > Spiegel <[email-masked]>; Tom Herbert 
>>> > <[email-masked]>
>>> > "In-situ OAM (IOAM) records OAM information within the packet 
>>> > while the packet traverses a particular network domain.  The term "in-situ"
>>> > UDP port numbers do not have global meaning. An intermediate 
>>> > device may very well see UDP packets destined to port 6081 that 
>>> > are not actually Geneve. This scenario is discussed in RFC7605:
>>> >
>>> >>> >> * Efficiency: IOAM adds data to live user traffic. As such, 
>>> >>> >> an encapsulation needs to be as efficient as possible. The “OOAM header”
>>> >>> >> is 8
>>> >>> >> header approach would add an unnecessary overhead of 4 bytes 
>>> >>> >> – which is significant.
>>> >>> Greg,
>>> >>>
>>> >>> I'm missing something here. I looked at the drafts you 
>>> >>> referenced and each of them looks like the overhead for OAM is 
>>> >>> greater that four bytes. In each there is some overhead 
>>> >>> equivalent to type/length, for instance in Geneve four bytes are 
>>> >>> needed for option class, type, and length. Unless the the OAM 
>>> >>> data is zero length, I don't see how this adds up to only four bytes of overhead.
>>> >>
>>> >>
>>> >>> >> Despite the question being asked, the recent thread on OOAM 
>>> >>> > I'm updating the OOAM Header draft and along with cleaning 
>>> >>> > nits will update reference to GUE. I think that the list and 
>>> >>> > the statemnt are quite clear in identifying the scope of 
>>> >>> > networks that may benefit from using not only common OOAM 
>>> >>> > Header but common OOAM mechanisms, e.g. Echo Request/Reply.
>>> >>> >

---

From: Tom Herbert <[email-masked]>
Date: Thu, 19 Apr 2018 08:43:32 -0700

Frank,

It's not a matter of ignoring IPv4, it's that IPv4 doesn't have the
capabilities that seem to be needed. Unless the intent is to resurrect
IPv4 options, there is no way to add ippm to IP packets other than by
encapsulation. That means that the cases I listed above that don't
involve encapsulation don't exist in IPv4. Also, again assuming IPv4
options are off the table, there is no way in IPv4 to perform ippm
hop-by-hop measurement since there is no allowance in IP that
intermediate devices can change IP payloads (see the UDP port number
misinterpretation problem for example). Maybe it's time to sunset IPv4
:-)

As for "popular encapsulations", I still don't see any feasible way to
extend IPIP for ippm. Also, I believe the attempt to make GRE
extensible as described in draft-weis-ippm-ioam-gre is problematic in
several ways-- I will send comments on that draft.

Tom

---

From: John Lemon <[email-masked]>
Date: Thu, 19 Apr 2018 09:09:22 -0700

I never saw a response to the request for a pointer to an OOAM
implementation, so I assume none exist.

I've seen several good arguments for why the existing IOAM implementation,
for which several implementations exist, meets the needs for IOAM.

I think it is time to put to bed the request to examine merging OOAM and
IOAM. Let's move forward with IOAM and not hold it up.

Respectfully, John

---

From: Greg Mirsky <[email-masked]>
Date: Thu, 19 Apr 2018 09:15:09 -0700

Hi Frank, et. al,
we have a very good discussion, thank you. I have a question and appreciate
your consideration:

   - encapsulation documents refer to IOAM HDR, its length is reflected in
   the field labeled either Length or IOAM HDR len. But I cannot find the
   definition of IOAM HDR. What is the IOAM HDR?

Regards,
Greg

---

From: Greg Mirsky <[email-masked]>
Date: Thu, 19 Apr 2018 09:30:59 -0700

Hi John,
I don't argue with the importance of interoperable implementations (though
early implementations accept the risk of non-compliance with the final
specification, for example, SFC NSH). At the same time, I don't think that
mere fact of existing implementation should cancel discussion of technical
characteristics of the proposed approaches to hybrid OAM.

Regards,
Greg

On Thu, Apr 19, 2018 at 9:09 AM, John Lemon <[email-masked]> wrote:

>> see https://datatracker.ietf.org/meeting/100/materials/slides-10
>> 0-hackathon-sessa-in-situ-oam-ioam - where we’ve shown IPv6 and
>> thank you for sharing your points. Please find my notes in-line and
>> tagged GIM>>. I believe that this is very much relevant to work of other
>> working groups that directly work on the overlay encapsulations in the
>> center of the discussion and hence I've added them to the list. Hope we'll
>> have more opinions to reach the conclusion that is acceptable to all.
>>
>>
>>
>> Regards,
>>
>> Greg
>>
>>
>>
>> could be leveraged.  After carefully considering
>> draft-ooamdt-rtgwg-ooam-header, I came to the conclusion that the “OOAM
>> header” does not meet the needs of IOAM:
>>

---

From: "Frank Brockners (fbrockne)" <[email-masked]>
Date: Thu, 19 Apr 2018 16:54:26 +0000

Hi Greg,

good catch – there is a bit of loose language in some of the drafts. We’ll make things crisper in the next rev. Note that there is no generic “IOAM header” but that definition is always within the context of a particular encapsulation protocol. draft-weis-ippm-ioam-gre-00 already has a definition of the IOAM header (for GRE) – see section 3. For the other drafts, we use language like “The IOAM related fields in VXLAN-GPE are defined as follows” or “The fields related to the encapsulation of IOAM data fields in Geneve are defined as follows”, i.e. the information that is required to perform the encapsulation into the parent protocol, along with the actual IOAM data fields. Moving forward, we can be crisper and split things into an “encapsulation dependent part” and a “data part”.

Frank

Sent: Donnerstag, 19. April 2018 18:15

Hi Frank, et. al,
we have a very good discussion, thank you. I have a question and appreciate your consideration:

  *   encapsulation documents refer to IOAM HDR, its length is reflected in the field labeled either Length or IOAM HDR len. But I cannot find the definition of IOAM HDR. What is the IOAM HDR?

Regards,
Greg

On Wed, Apr 11, 2018 at 3:02 AM, Frank Brockners (fbrockne) <[email-masked]<mailto:[email-masked]>> wrote:

Regards, Frank

---

From: Greg Mirsky <[email-masked]>
Date: Thu, 19 Apr 2018 10:26:37 -0700

Hi Frank,
thank you for your expedient response. Yes, clarification and consistent
terminology, of course as different encapsulations allow that, will help.
What I'm looking through the iOAM encapsulation drafts is the answer to
this question How a system that is not using iOAM can get to the data
payload that follows the iOAM message? Is there the field in the iOAM shim
that allows the system to skip over the iOAM message (by iOAM message I
mean iOAM shim and iOAM data)? Would such system be required to parse other
than iOAM shim constructs? I couldn't find this scenario being discussed in
any of iOAM encapsulation drafts. Have I missed it?

Regards,
Greg

On Thu, Apr 19, 2018 at 9:54 AM, Frank Brockners (fbrockne) <
[email-masked]> wrote:

> Hi Greg,
>
>
>
> good catch – there is a bit of loose language in some of the drafts. We’ll
> make things crisper in the next rev. Note that there is no generic “IOAM
> header” but that definition is always within the context of a particular
> encapsulation protocol. draft-weis-ippm-ioam-gre-00 already has a
> definition of the IOAM header (for GRE) – see section 3. For the other
> drafts, we use language like “The IOAM related fields in VXLAN-GPE are
> defined as follows” or “The fields related to the encapsulation of IOAM
> data fields in Geneve are defined as follows”, i.e. the information that is
> required to perform the encapsulation into the parent protocol, along with
> the actual IOAM data fields. Moving forward, we can be crisper and split
> things into an “encapsulation dependent part” and a “data part”.
>
>
>
> Frank
>
>
>
> *Sent:* Donnerstag, 19. April 2018 18:15
>    - encapsulation documents refer to IOAM HDR, its length is reflected
>    in the field labeled either Length or IOAM HDR len. But I cannot find the

---

From: John Lemon <[email-masked]>
Date: Thu, 19 Apr 2018 15:53:37 -0700

Hi Greg,

I never stated "that mere fact of existing implementation should cancel
discussion of technical characteristics of the proposed approaches to
hybrid OAM". I just noted implementation status as AN important thing to
consider. I also noted that "I've seen several good arguments for why the
existing IOAM implementation [...] meets the needs for IOAM." I'm not
trying to end discussion of the technical characteristics. I'm stating that
I believe that it has been well argued that IOAM is mature enough that it
is clear that it is sufficiently different from OOAM as to not share the
same header.

I hope that clarifies my intent.

Regards, John

On Thu, Apr 19, 2018 at 9:30 AM, Greg Mirsky <[email-masked]> wrote:

>> I've seen several good arguments for why the existing IOAM
>> implementation, for which several implementations exist, meets the needs
>> for IOAM.
>>
>>> GIM>> May I ask which encapsulations supported by the implementations
>>> you refer to. Until very recently all iOAM proposals were to use meta-data
>>> TLV in, e.g. Geneve and NSH. And if these or some of these implementations
>>> * Scope: It isn’t entirely clear to which protocols the OOAM header
>>> would ultimately apply to. The way the OOAM header is defined, OOAM uses a
>>> 8-bit field for “Next Prot”, the next protocol. Some protocols that IOAM
>>> data needs to be encapsulated into use 16-bits for their next protocol code

---

From: "Frank Brockners (fbrockne)" <[email-masked]>
Date: Fri, 20 Apr 2018 07:15:51 +0000

Hi Greg,

it depends on the individual encapsulation, i.e. the parent protocol used. IOAM data encapsulation depends on what the parent protocol offers as encapsulation mechanism. As such, you’d depend on the procedures used by the parent protocol.
Let’s consider two examples Geneve and NSH: For Geneve (draft-brockners-ippm-ioam-geneve-00) we use the “option class” mechanism to carry IOAM data, i.e. Geneve’s mechanism to carry meta data. Getting to the payload in a setup with IOAM data within the Geneve header is no different than a Geneve setup without IOAM, assuming that the Geneve implementation supports Geneve option classes. Check out draft-ietf-nvo3-geneve for the Geneve header. For NSH the suggestion in draft-brockners-sfc-ioam-nsh-01 is to use the “NSH Next Protocol” mechanism in NSH (see e.g. RFC 8300, section 9.1.6). Re-specifying the encap mechanism of the base protocol in a specification which only leverages the base protocol will do more harm than good. It should be avoided because it could only lead to confusion. All the IOAM encap drafts clearly reference the parent protocol encap mechanism – so an implementer will naturally refer to the base specification.

All that said: What are you trying to point out? It seems that you’re pickup up the discussion about the pros and cons on whether to use protocol meta-data or the “next header” approach. The SFC WG discussed this at length in the last meeting in London and you actively participated in it. The “next header” approach allows for a more efficient implementation in hardware (fewer nested structures/lookups) at the expense of requiring the each node to have a basic understanding of IOAM (at least the type and length fields – so that you could skip past it), whereas the “leverage meta-data type-2” approach leads to more complicated lookup operations (the location of MD Type 2 isn’t fixed) as well as constrain the amount of data to be carried to 256 octets. See also section 4.1 in draft-brockners-sfc-ioam-nsh-01.
You remember that from the “hum” in the room that the chairs initiated (which still needs confirmation on the list – which we do right now), there seems to be a preference to either go with the “next header approach” (as per draft-brockners-sfc-ioam-nsh-01) or to document the next header approach and the MD-Type2 approach. We do care about efficient implementation – this is what we learned from several IOAM implementations by now, and which is why we have quite a few authors from companies which drive silicon implementations on the IOAM drafts – see also John Lemon’s email yesterday.

We’d greatly appreciate thoughts from other SFC WG members on their preference of using either the “NSH next protocol” approach or the “MD Type2” meta-data based approach.

Thanks,
Frank

Sent: Donnerstag, 19. April 2018 19:27

Hi Frank,
thank you for your expedient response. Yes, clarification and consistent terminology, of course as different encapsulations allow that, will help. What I'm looking through the iOAM encapsulation drafts is the answer to this question How a system that is not using iOAM can get to the data payload that follows the iOAM message? Is there the field in the iOAM shim that allows the system to skip over the iOAM message (by iOAM message I mean iOAM shim and iOAM data)? Would such system be required to parse other than iOAM shim constructs? I couldn't find this scenario being discussed in any of iOAM encapsulation drafts. Have I missed it?

Regards,
Greg

On Thu, Apr 19, 2018 at 9:54 AM, Frank Brockners (fbrockne) <[email-masked]<mailto:[email-masked]>> wrote:
Hi Greg,

Frank

Hi Frank, et. al,

Regards,
Greg

Regards, Frank

---

From: "Carlos Pignataro (cpignata)" <[email-masked]>
Date: Fri, 20 Apr 2018 19:03:21 +0000

Tom,

On Apr 17, 2018, at 10:22 AM, Tom Herbert <[email-masked]<mailto:[email-masked]>> wrote:

Hi Tianran,

Frank,

In that case why not just use a hop-by-hop option for measuring the
underlay and a destination option for measuring the overlay? The
advantage is that this works _any_ IP encapsulation method or any IP
protocol for that matter.

Because you want to instrument the layer that you want to measure.
Because there’s cases with more unnatural layering where there’s a desire to correlate and compare measurements across layers (in a way in which, for example, the Service layer is tested in a service chaining scenario, not the IPv6 hop-by-hop.
Because different topologies expose different Hops and IPv6 HBH goes by the IPv6 node topology.
Because not everything is IPv6, and because you have cases of IPv6 over something as well.
Those are quick ones that come to mind.

I don't believe adding ippm to every

Engineering is about trade-offs. If you want to measure Geneve, there are limitations. But instead of trying to prove why it does not work, I’ll point to working demos of where it does — many of which on different HW/SW and encaps, shown at various IETF events.

Thanks,

— Carlos Pignataro

Tom

Frank

-----Original Message-----
From: Tianran Zhou <[email-masked]<mailto:[email-masked]>>
Sent: Dienstag, 17. April 2018 03:18
To: Tom Herbert <[email-masked]<mailto:[email-masked]>>
Cc: Shwetha Bhandari (shwethab) <[email-masked]<mailto:[email-masked]>>; Frank Brockners (fbrockne) <[email-masked]<mailto:[email-masked]>>; Mickey Spiegel <[email-masked]<mailto:[email-masked]>>; NVO3 <[email-masked]<mailto:[email-masked]>>; Service Function Chaining IETF list <[email-masked]<mailto:[email-masked]>>; IETF IPPM WG <[email-masked]<mailto:[email-masked]>>
Subject: RE: [ippm] [Int-area] encapsulation of IOAM data in various protocols - follow up from WG discussion in London

Tianran
-----Original Message-----
From: Tom Herbert [mailto:[email-masked]]
Sent: Monday, April 16, 2018 10:40 PM
To: Tianran Zhou <[email-masked]<mailto:[email-masked]>>
Cc: Shwetha Bhandari (shwethab) <[email-masked]<mailto:[email-masked]>>; Frank Brockners
(fbrockne) <[email-masked]<mailto:[email-masked]>>; Mickey Spiegel
<[email-masked]<mailto:[email-masked]>>; NVO3 <[email-masked]<mailto:[email-masked]>>; int-area
<[email-masked]<mailto:[email-masked]>>; Service Function Chaining IETF list
<[email-masked]<mailto:[email-masked]>>; IETF IPPM WG <[email-masked]<mailto:[email-masked]>>
Subject: Re: [ippm] [Int-area] encapsulation of IOAM data in various
protocols - follow up from WG discussion in London

On Mon, Apr 16, 2018 at 6:31 AM, Tianran Zhou <[email-masked]<mailto:[email-masked]>> wrote:
Hi Shwetha,

Hi Tianran,

Tom

Tianran

发件人： Shwetha Bhandari (shwethab)
收件人： Tianran Zhou<[email-masked]<mailto:[email-masked]>>;Frank Brockners
(fbrockne)<[email-masked]<mailto:[email-masked]>>;Mickey
Spiegel<[email-masked]<mailto:[email-masked]>>;Tom
Herbert<[email-masked]<mailto:[email-masked]>>
抄送： NVO3<[email-masked]<mailto:[email-masked]>>;int-area<[email-masked]<mailto:[email-masked]>>;Service Function
Chaining IETF list<[email-masked]<mailto:[email-masked]>>;IETF IPPM WG<[email-masked]<mailto:[email-masked]>>
主题： Re: [ippm] [Int-area] encapsulation of IOAM data in various
protocols - follow up from WG discussion in London
时间： 2018-04-16 18:17:01

Hi Tianran,

Thanks,

Shwetha

From: ippm <[email-masked]<mailto:[email-masked]>> on behalf of Tianran Zhou
<[email-masked]<mailto:[email-masked]>>
Date: Monday, April 16, 2018 at 2:36 PM
To: "Frank Brockners (fbrockne)" <[email-masked]<mailto:[email-masked]>>, Mickey
Spiegel <[email-masked]<mailto:[email-masked]>>, Tom Herbert
<[email-masked]<mailto:[email-masked]>>
Cc: NVO3 <[email-masked]<mailto:[email-masked]>>, "[email-masked]<mailto:[email-masked]>" <[email-masked]<mailto:[email-masked]>>,
Service Function Chaining IETF list <[email-masked]<mailto:[email-masked]>>, IETF IPPM WG
<[email-masked]<mailto:[email-masked]>>
Subject: Re: [ippm] [Int-area] encapsulation of IOAM data in various
protocols - follow up from WG discussion in London

Hi Frank,

  +--rw ioam

     +--rw ioam-profiles

        +--rw enabled?        boolean

           +--rw filter

           |  ...

           |  ...

           |  ...

              ...

Tianran

From: Frank Brockners (fbrockne) [mailto:[email-masked]]
Sent: Monday, April 16, 2018 4:51 PM
To: Tianran Zhou <[email-masked]<mailto:[email-masked]>>; Mickey Spiegel
<[email-masked]<mailto:[email-masked]>>; Tom Herbert <[email-masked]<mailto:[email-masked]>>
Cc: NVO3 <[email-masked]<mailto:[email-masked]>>; [email-masked]<mailto:[email-masked]>; Service Function
Chaining IETF list <[email-masked]<mailto:[email-masked]>>; IETF IPPM WG <[email-masked]<mailto:[email-masked]>>
Subject: RE: [ippm] [Int-area] encapsulation of IOAM data in various
protocols - follow up from WG discussion in London

Hi Tianran,

Frank

From: Tianran Zhou <[email-masked]<mailto:[email-masked]>>
Sent: Montag, 16. April 2018 10:37
To: Frank Brockners (fbrockne) <[email-masked]<mailto:[email-masked]>>; Mickey Spiegel
<[email-masked]<mailto:[email-masked]>>; Tom Herbert <[email-masked]<mailto:[email-masked]>>
Cc: NVO3 <[email-masked]<mailto:[email-masked]>>; [email-masked]<mailto:[email-masked]>; Service Function
Chaining IETF list <[email-masked]<mailto:[email-masked]>>; IETF IPPM WG <[email-masked]<mailto:[email-masked]>>
Subject: RE: [ippm] [Int-area] encapsulation of IOAM data in various
protocols - follow up from WG discussion in London

Hi Frank,

Tianran

From: ippm [mailto:[email-masked]] On Behalf Of Frank
Brockners
(fbrockne)
Sent: Monday, April 16, 2018 3:08 PM
To: Mickey Spiegel <[email-masked]<mailto:[email-masked]>>; Tom Herbert
<[email-masked]<mailto:[email-masked]>>
Cc: NVO3 <[email-masked]<mailto:[email-masked]>>; [email-masked]<mailto:[email-masked]>; Service Function
Chaining IETF list <[email-masked]<mailto:[email-masked]>>; IETF IPPM WG <[email-masked]<mailto:[email-masked]>>
Subject: Re: [ippm] [Int-area] encapsulation of IOAM data in various
protocols - follow up from WG discussion in London

Tom,

Frank

From: ippm <[email-masked]<mailto:[email-masked]>> On Behalf Of Mickey Spiegel
Sent: Freitag, 13. April 2018 20:22
To: Tom Herbert <[email-masked]<mailto:[email-masked]>>
Cc: NVO3 <[email-masked]<mailto:[email-masked]>>; [email-masked]<mailto:[email-masked]>; Service Function
Chaining IETF list <[email-masked]<mailto:[email-masked]>>; IETF IPPM WG <[email-masked]<mailto:[email-masked]>>
Subject: Re: [ippm] [Int-area] encapsulation of IOAM data in various
protocols - follow up from WG discussion in London

Tom,

Mickey,

VXLAN rather than NSH.

Mickey

Thanks,
Tom

Tom,

On Thu, Apr 12, 2018 at 9:54 AM, Greg Mirsky
<[email-masked]<mailto:[email-masked]>>
wrote:
Hi Frank,
thank you for sharing your points. Please find my notes in-line
and tagged
GIM>>. I believe that this is very much relevant to work of
GIM>>other
working
groups that directly work on the overlay encapsulations in the
center of the discussion and hence I've added them to the list.
Hope we'll have more opinions to reach the conclusion that is
acceptable to all.

Regards,
Greg

the conclusion that the “OOAM header” does not meet the needs
of
IOAM:

encapsulation needs to be as efficient as possible. The “OOAM header”

Tom

Regards, Frank

_______________________________________________
sfc mailing list
[email-masked]<mailto:[email-masked]>
https://www.ietf.org/mailman/listinfo/sfc

---

From: Tom Herbert <[email-masked]>
Date: Sat, 21 Apr 2018 08:05:05 -0700

On Fri, Apr 20, 2018 at 12:03 PM, Carlos Pignataro (cpignata)
<[email-masked]> wrote:
> Tom,
>
> On Apr 17, 2018, at 10:22 AM, Tom Herbert <[email-masked]> wrote:
>
> Tom's note already includes the hint: You'll add IOAM data to the
> protocol/layer that you're interested in monitoring. Again using Geneve over
> IPv6 as an example:
> * If you're interested in the overlay, i.e. Geneve (e.g. timestamping the
> packet when it enters and exists the tunnel) - you'd add IOAM data to Geneve
> understand which path packets take in the v6 network) - you'd add IOAM data
> to IPv6
> Because there’s cases with more unnatural layering where there’s a desire to
> correlate and compare measurements across layers (in a way in which, for
> example, the Service layer is tested in a service chaining scenario, not the
> IPv6 hop-by-hop.
Carlos,

Please see my other email that details some use cases that shows
destination options are functionally equivalent to ippm in
encapsulation, and also my comments that the IPv6 has superior
capabilities to cover in-situ ippm requirements (in particular that IP
options are the _only_ protocol conformant means for intermediate
nodes to change IP payloads needed for IOAM tracing).

I don't have a general issue with supporting ippm in encapsulation,
but I do think this should be viewed as legacy support. Note there is
no concept of segment routing in IPv4, they are blazing forward only
on IPv6 so it is reasonable to take this view. Personally, I don't
think this is a disadvantage to SR. IPv6 does have more capabilities
than IPv4 and we're now seeing protocols that will take advantage of
those. Features like this are good motivation for moving to IPv6,
which in the long run is good for the Internet!

Tom

> Engineering is about trade-offs. If you want to measure Geneve, there are
> limitations. But instead of trying to prove why it does not work, I’ll point
> to working demos of where it does — many of which on different HW/SW and
> encaps, shown at various IETF events.
>
> Thanks,
>
> — Carlos Pignataro
>
> Tom
>
> "Layering: If several encapsulation protocols (e.g., in case of tunneling)
> are stacked on top of each other, IOAM data-records could be present at
> every layer.  The behavior follows the ships-in-the-night model."
>
> Given the discussion here, we'll add some additional text in the next
> revision to make things crisper (e.g. adding an example might help).
>
> Frank
>
> <[email-masked]>; NVO3 <[email-masked]>; Service Function
> than Geneve and may be misinterpreted. If a misinterpreted packet is changed
> (like ippm data is written) then that would be systematic silent data
> corruption.
>
> [email-masked]

---

From: "Carlos Pignataro (cpignata)" <[email-masked]>
Date: Sun, 22 Apr 2018 09:21:25 +0000

Hi Tom,

I agree that using IOAM in IPv6 both e2e and hbh is a powerful and useful combo!

My point, sorry if I was not clear, is that an “SFC Hop” does not correspond to a transport encapsulation hop, and that IOAM can be in-situ’ed to the encapsulation that realizes the (service, overlay, otherwise higher) topology (which can be IPv6 natively or something else as well)

Thanks,

Thumb typed by Carlos Pignataro.
Excuze typofraphicak errows

> On Apr 21, 2018, at 11:05, Tom Herbert <[email-masked]> wrote:
>

---

From: Tom Herbert <[email-masked]>
Date: Sun, 22 Apr 2018 09:28:18 -0700

Carlos,

AFACT, the intent is that nodes along the path of a packet containing
in-situ ippm data may modify the ippm data as described in
draft-ietf-ippm-ioam-data. Your comment confirms my belief that part
of the intent is that intermediate nodes, specifically nodes that are
not addressed by the destination address of a packet, may also modify
ippm data.

If this is correct, then I understand how this process could work
correctly with hop-by-hop options. However, I don't understand how
this can work correclty with encapsulation where the ippm data is
within the encapsulation. IP has no allowance for intermediate nodes
to modify transport payloads. For example, if UDP payloads are being
modified in the network, then this introduces the possibility of
silent  corruption when the port number is misinterpreted.

Thanks,
Tom

---

From: "Carlos Pignataro (cpignata)" <[email-masked]>
Date: Mon, 23 Apr 2018 00:00:25 +0000

Hi Tom,

A quick top-post response: I believe you are making assumptions about how this works in a multiplayer environment, and about which nodes can modify what. 

Instead of email, let us (like Frank had also said) expand on a tighter description and use case example in the draft itself, Section 3, and close this loop that way. 

> On Apr 22, 2018, at 11:28, Tom Herbert <[email-masked]> wrote:
>