Subject: [ippm] Working Group Last Call: draft-ietf-ippm-ioam-data

From: Tommy Pauly <[email-masked]>
Date: Tue, 07 Jan 2020 08:47:33 -0800

Hello IPPM,

Happy New Year! This email starts a working group last call for "Data Fields for In-situ OAM" (https://datatracker.ietf.org/doc/draft-ietf-ippm-ioam-data/).

The last call will end on Tuesday, January 21. Please reply to [email-masked] with your reviews and comments. Please indicate whether you think this document is ready for publication.

Best,
Tommy (as co-chair)

---

From: Haoyu Song <[email-masked]>
Date: Tue, 7 Jan 2020 22:25:36 +0000

IPPM and IOAM authors,

I have a question about the variable length opaque data field in the trace option. Does that mean each node may add a variable length field (i.e., use a different scheme ID)?

     "When this field is part of the data field but a node populating
      the field has no opaque state data to report, the Length must be
      set to 0 and the Schema ID must be set to 0xFFFFFF to mean no
      schema."

>From the above description, it seems so.  Then there's a problem. If each node's data has different size, then one can't guarantee that the RemainingLen is exactly 0 when the data overflow happens.

Another issue:

    "If IOAM-Trace-Type bit 22 is not set, then NodeLen specifies the
      actual length added by each node.  If IOAM-Trace-Type bit 22 is
      set, then the actual length added by a node would be (NodeLen +
      Opaque Data Length)."

I think the "actual length added by a node would be (NodeLen+ size of opaque state snapshot field). Because the opaque data is only a part of the opaque state snapshot field, and  (opaque data length + 4 byte)=size of opaque state snapshot field.  Some other places need similar clarification.

Haoyu

From: ippm <[email-masked]> On Behalf Of Tommy Pauly
Sent: Tuesday, January 07, 2020 8:48 AM
To: IETF IPPM WG <[email-masked]>
Subject: [ippm] Working Group Last Call: draft-ietf-ippm-ioam-data

Hello IPPM,

Happy New Year! This email starts a working group last call for "Data Fields for In-situ OAM" (https://datatracker.ietf.org/doc/draft-ietf-ippm-ioam-data/<https://nam03.safelinks.protection.outlook.com/?url=https%3A%2F%2Fdatatracker.ietf.org%2Fdoc%2Fdraft-ietf-ippm-ioam-data%2F&data=02%7C01%7Chaoyu.song%40futurewei.com%7C62dfcd2f485d4a27f58208d79391677f%7C0fee8ff2a3b240189c753a1d5591fedc%7C1%7C0%7C637140125044139853&sdata=Sf3SxjE%2FqK3X760G8c0Tfemu%2FPiqpkpIVHswudQWDTc%3D&reserved=0>).

The last call will end on Tuesday, January 21. Please reply to [email-masked]<mailto:[email-masked]> with your reviews and comments. Please indicate whether you think this document is ready for publication.

Best,
Tommy (as co-chair)

---

From: Barak Gafni <[email-masked]>
Date: Tue, 21 Jan 2020 07:33:19 +0000

Hi,
Few comments to consider:

  1.  Queue depth format - In order to align the standardization of units across the route, as done for time measurements for example, I would suggest to direct implementations towards reporting Bytes, rather than keeping it open for interpretation by the implementors.
  2.  Additional measurements to consider - amount of Bytes transmitted by a port, rate of transmission of a port.

Thanks,
Barak

From: ippm <[email-masked]> On Behalf Of Tommy Pauly
Sent: Tuesday, January 7, 2020 8:48 AM
To: IETF IPPM WG <[email-masked]>
Subject: [ippm] Working Group Last Call: draft-ietf-ippm-ioam-data

Hello IPPM,

Happy New Year! This email starts a working group last call for "Data Fields for In-situ OAM" (https://datatracker.ietf.org/doc/draft-ietf-ippm-ioam-data/<https://eur03.safelinks.protection.outlook.com/?url=https%3A%2F%2Fdatatracker.ietf.org%2Fdoc%2Fdraft-ietf-ippm-ioam-data%2F&data=02%7C01%7Cgbarak%40mellanox.com%7Cc22db9cdb94f474ad9aa08d793915782%7Ca652971c7d2e4d9ba6a4d149256f461b%7C0%7C0%7C637140124776780825&sdata=K%2BYMdN2sPBGrzJkqUnHLybaMnxCWe6BONvEsMrWBpv4%3D&reserved=0>).

Best,
Tommy (as co-chair)

---

From: "Frank Brockners (fbrockne)" <[email-masked]>
Date: Tue, 21 Jan 2020 13:40:46 +0000

Haoyu,

Thanks. Please see inline ("...FB").

From: ippm <[email-masked]> On Behalf Of Haoyu Song
Sent: Dienstag, 7. Januar 2020 23:26
To: Tommy Pauly <tpauly=[email-masked]>; IETF IPPM WG <[email-masked]>
Subject: Re: [ippm] Working Group Last Call: draft-ietf-ippm-ioam-data

IPPM and IOAM authors,

>From the above description, it seems so.  Then there's a problem. If each node's data has different size, then one can't guarantee that the RemainingLen is exactly 0 when the data overflow happens.

...FB: Could you detail the problem a bit more: Why would the remaining length need to be exactly 0 in case of a data overflow? If RemainingLen < (NodeLen + sizeof(opaque snapshot)) in 4 octet units, then a node would not add data.

Another issue:

I think the "actual length added by a node would be (NodeLen+ size of opaque state snapshot field). Because the opaque data is only a part of the opaque state snapshot field, and  (opaque data length + 4 byte)=size of opaque state snapshot field.  Some other places need similar clarification.
...FB: Thanks. It makes sense to harmonize the description and at the same time, make it more specific - i.e. match it  to what is already used in section 4.4.1, i.e. "NodeLen + sizeof(opaque snapshot)" in 4 octet units.
Cheers, Frank

Haoyu

From: ippm <[email-masked]<mailto:[email-masked]>> On Behalf Of Tommy Pauly
Sent: Tuesday, January 07, 2020 8:48 AM
To: IETF IPPM WG <[email-masked]<mailto:[email-masked]>>
Subject: [ippm] Working Group Last Call: draft-ietf-ippm-ioam-data

Hello IPPM,

The last call will end on Tuesday, January 21.. Please reply to [email-masked]<mailto:[email-masked]> with your reviews and comments. Please indicate whether you think this document is ready for publication.

Best,
Tommy (as co-chair)

---

From: "Frank Brockners (fbrockne)" <[email-masked]>
Date: Tue, 21 Jan 2020 13:53:36 +0000

Barak,

From: ippm <[email-masked]> On Behalf Of Barak Gafni
Sent: Dienstag, 21. Januar 2020 08:33
To: Tommy Pauly <tpauly=[email-masked]>; IETF IPPM WG <[email-masked]>
Subject: Re: [ippm] Working Group Last Call: draft-ietf-ippm-ioam-data

Hi,

  1.  Queue depth format - In order to align the standardization of units across the route, as done for time measurements for example, I would suggest to direct implementations towards reporting Bytes, rather than keeping it open for interpretation by the implementors.
...FB: As you know, the original rationale used throughout IOAM is to avoid forcing the data producer (e.g. the forwarding ASIC) to translate a counter into a specific unit, which could be costly. One would rather do the transformation on the receiver of the data than on the sender of the data. At this point you'd probably argue that for time, we also specify the format/unit - though for time, we have 3 options available, and one of which is expected to be supported by the source natively, hence the "exception".

  1.  Additional measurements to consider - amount of Bytes transmitted by a port, rate of transmission of a port.
...FB: The data fields currently defined in the draft are considered a "baseline" - i.e. a set that everyone agrees with. We all know there have been proposals for other data fields in the past - but those proposals never got broader momentum. One can always leverage the opaque state snapshot to carry deployment specific data. So in order to add those two additional data fields to the draft at this stage, we'd need to get a good understanding of how common the need for the two additional data fields is. Who else would see a need? What would be the use-cases?

Thanks, Frank

Thanks,
Barak

Hello IPPM,

Best,
Tommy (as co-chair)

---

From: Haoyu Song <[email-masked]>
Date: Tue, 21 Jan 2020 17:15:35 +0000

Inline

From: Frank Brockners (fbrockne) <[email-masked]>
Sent: Tuesday, January 21, 2020 5:41 AM
To: Haoyu Song <[email-masked]>; Tommy Pauly <tpauly=[email-masked]>; IETF IPPM WG <[email-masked]>
Subject: RE: [ippm] Working Group Last Call: draft-ietf-ippm-ioam-data

Haoyu,

From: ippm <[email-masked]<mailto:[email-masked]>> On Behalf Of Haoyu Song
Sent: Dienstag, 7. Januar 2020 23:26
To: Tommy Pauly <tpauly=[email-masked]<mailto:tpauly=[email-masked]>>; IETF IPPM WG <[email-masked]<mailto:[email-masked]>>
Subject: Re: [ippm] Working Group Last Call: draft-ietf-ippm-ioam-data

IPPM and IOAM authors,

[HS] According to the draft text,  "RemainingLen:  7-bit unsigned integer.  This field specifies the data
      space in multiples of 4-octets remaining for recording the node
      data, before the node data list is considered to have overflowed.
      When RemainingLen reaches 0, nodes are no longer allowed to add
      node data. ", so I assume that the Remaining length of 0 means no more data can be added.

      Also, "NodeLen"  is fixed but the size of snapshot is not, which makes each node probably add different number of bytes.
      This will make it difficult to find the location for new data insertion in the pre-allocated trace mode.

Another issue:

Haoyu

Hello IPPM,

Happy New Year! This email starts a working group last call for "Data Fields for In-situ OAM" (https://datatracker.ietf.org/doc/draft-ietf-ippm-ioam-data/<https://nam03.safelinks.protection.outlook.com/?url=https%3A%2F%2Fdatatracker.ietf.org%2Fdoc%2Fdraft-ietf-ippm-ioam-data%2F&data=02%7C01%7Chaoyu.song%40futurewei.com%7C4f471415b6d84a2f65f508d79e778839%7C0fee8ff2a3b240189c753a1d5591fedc%7C1%7C0%7C637152108543542112&sdata=sDmCSLe4dQlxFEwqiPdfsWTWdf5q0MPSRwIL6lcDxZs%3D&reserved=0>).

Best,
Tommy (as co-chair)

---

From: Barak Gafni <[email-masked]>
Date: Tue, 21 Jan 2020 19:08:53 +0000

PSB under [BG]

Thanks,
Barak

From: Frank Brockners (fbrockne) <[email-masked]>
Sent: Tuesday, January 21, 2020 5:54 AM
To: Barak Gafni <[email-masked]>; Tommy Pauly <tpauly=[email-masked]>; IETF IPPM WG <[email-masked]>
Subject: RE: [ippm] Working Group Last Call: draft-ietf-ippm-ioam-data

Barak,

From: ippm <[email-masked]<mailto:[email-masked]>> On Behalf Of Barak Gafni
Sent: Dienstag, 21. Januar 2020 08:33
To: Tommy Pauly <tpauly=[email-masked]<mailto:tpauly=[email-masked]>>; IETF IPPM WG <[email-masked]<mailto:[email-masked]>>
Subject: Re: [ippm] Working Group Last Call: draft-ietf-ippm-ioam-data

Hi,

  1.  Queue depth format - In order to align the standardization of units across the route, as done for time measurements for example, I would suggest to direct implementations towards reporting Bytes, rather than keeping it open for interpretation by the implementors.
...FB: As you know, the original rationale used throughout IOAM is to avoid forcing the data producer (e.g. the forwarding ASIC) to translate a counter into a specific unit, which could be costly. One would rather do the transformation on the receiver of the data than on the sender of the data. At this point you'd probably argue that for time, we also specify the format/unit - though for time, we have 3 options available, and one of which is expected to be supported by the source natively, hence the "exception".
[BG] Agree, forcing this reporting method may be too much for some implementation that we don't want to prevent from supporting IOAM. Hence, my suggestion is to use SHOULD, which can drive better interoperability going forward without prevent current implementations from being compatible.

  1.  Additional measurements to consider - amount of Bytes transmitted by a port, rate of transmission of a port.
...FB: The data fields currently defined in the draft are considered a "baseline" - i.e. a set that everyone agrees with. We all know there have been proposals for other data fields in the past - but those proposals never got broader momentum. One can always leverage the opaque state snapshot to carry deployment specific data. So in order to add those two additional data fields to the draft at this stage, we'd need to get a good understanding of how common the need for the two additional data fields is. Who else would see a need? What would be the use-cases?
[BG] Would be great to have the list further comment on that

Thanks, Frank

Thanks,
Barak

Hello IPPM,

Happy New Year! This email starts a working group last call for "Data Fields for In-situ OAM" (https://datatracker.ietf.org/doc/draft-ietf-ippm-ioam-data/<https://eur03.safelinks.protection.outlook.com/?url=https%3A%2F%2Fdatatracker.ietf.org%2Fdoc%2Fdraft-ietf-ippm-ioam-data%2F&data=02%7C01%7Cgbarak%40mellanox.com%7C03abe451ab064dba268608d79e7952b8%7Ca652971c7d2e4d9ba6a4d149256f461b%7C0%7C0%7C637152116230294920&sdata=V8DCZz%2ByMeocyFRBX6gSzYYd5n0EdO%2FZPGWqnkqHguc%3D&reserved=0>).

Best,
Tommy (as co-chair)

---

From: Mickey Spiegel <[email-masked]>
Date: Tue, 21 Jan 2020 14:30:04 -0800

Inline comments under [MS]

On Tue, Jan 21, 2020 at 11:09 AM Barak Gafni <[email-masked]> wrote:

> PSB under [BG]
>
>
>
> Thanks,
>
> Barak
>
>
>
> *From:* Frank Brockners (fbrockne) <[email-masked]>
> *Sent:* Tuesday, January 21, 2020 5:54 AM
> *To:* Barak Gafni <[email-masked]>; Tommy Pauly <tpauly=
> [email-masked]>; IETF IPPM WG <[email-masked]>
> *Subject:* RE: [ippm] Working Group Last Call: draft-ietf-ippm-ioam-data
>
>
>
> Barak,
>
>
>
> Thanks. Please see inline (“…FB”).
>
>
>
> *From:* ippm <[email-masked]> *On Behalf Of *Barak Gafni
> *Sent:* Dienstag, 21. Januar 2020 08:33
> *To:* Tommy Pauly <tpauly=[email-masked]>; IETF IPPM WG <
> [email-masked]>
> *Subject:* Re: [ippm] Working Group Last Call: draft-ietf-ippm-ioam-data
>
>
>
> Hi,
>
>    1. Queue depth format – In order to align the standardization of units
>    across the route, as done for time measurements for example, I would
>    suggest to direct implementations towards reporting Bytes, rather than
>    keeping it open for interpretation by the implementors.
>
> *…FB: As you know, the original rationale used throughout IOAM is to avoid
> forcing the data producer (e.g. the forwarding ASIC) to translate a counter
> into a specific unit, which could be costly. One would rather do the
> transformation on the receiver of the data than on the sender of the data.
> At this point you’d probably argue that for time, we also specify the
> format/unit – though for time, we have 3 options available, and one of
> which is expected to be supported by the source natively, hence the
> “exception”.*
>
> *[BG] Agree, forcing this reporting method may be too much for some
> implementation that we don’t want to prevent from supporting IOAM. Hence,
> my suggestion is to use SHOULD, which can drive better interoperability
> going forward without prevent current implementations from being
> compatible.*
>

[MS] There are several possible use cases for IOAM. Much of the initial
work has been motivated by flow monitoring use cases, where switches and
routers embed or export metadata relating to packets in a flow, at line
rate for massively scaled data centers. At the receiver, typically the
exported metadata is processed and analyzed in software. In such a
scenario, in order to enable IOAM to work across as wide a range of devices
as possible, it makes sense to use the design philosophy that Frank stated
above: Allow the data producer to generate metadata in whatever units it
wants, and normalize at the receiver where the exported metadata is
processed and analyzed. For such use cases, I believe that standardization
of units should not be a goal, not even as a recommendation.

[MS] There may be other use cases where standardization of units makes
sense. Rather than constrain all use cases, my suggestion would be to
define recommendations specific to such use cases, perhaps using the
concept of profiles that Tal has proposed.

>    1. Additional measurements to consider – amount of Bytes transmitted
>    by a port, rate of transmission of a port.
>
>
[MS] I support the addition of the amount of bytes transmitted by a port.
However, I wonder whether this should be the amount of bytes transmitted on
the entire port, or the amount of bytes transmitted using a specific queue
on the port?

Mickey

>
>    1.
>
> *…FB: The data fields currently defined in the draft are considered a
> “baseline” – i.e. a set that everyone agrees with. We all know there have
> been proposals for other data fields in the past – but those proposals
> never got broader momentum. One can always leverage the opaque state
> snapshot to carry deployment specific data. So in order to add those two
> additional data fields to the draft at this stage, we’d need to get a good
> understanding of how common the need for the two additional data fields is.
> Who else would see a need? What would be the use-cases?*
>
> *[BG] Would be great to have the list further comment on that*
>
>
>
> *Thanks, Frank*
>
>
>
> Thanks,
>
> Barak
>
>
>
> *From:* ippm <[email-masked]> *On Behalf Of *Tommy Pauly
> *Sent:* Tuesday, January 7, 2020 8:48 AM
> *To:* IETF IPPM WG <[email-masked]>
> *Subject:* [ippm] Working Group Last Call: draft-ietf-ippm-ioam-data
>
>
>
> Hello IPPM,
>
>
>
> Happy New Year! This email starts a working group last call for "Data
> Fields for In-situ OAM" (
> https://datatracker.ietf.org/doc/draft-ietf-ippm-ioam-data/
> <https://eur03.safelinks.protection.outlook.com/?url=https%3A%2F%2Fdatatracker.ietf.org%2Fdoc%2Fdraft-ietf-ippm-ioam-data%2F&data=02%7C01%7Cgbarak%40mellanox.com%7C03abe451ab064dba268608d79e7952b8%7Ca652971c7d2e4d9ba6a4d149256f461b%7C0%7C0%7C637152116230294920&sdata=V8DCZz%2ByMeocyFRBX6gSzYYd5n0EdO%2FZPGWqnkqHguc%3D&reserved=0>
> ).
>
>
>
> The last call will end on *Tuesday, January 21*... Please reply to
> [email-masked] with your reviews and comments. Please indicate whether you
> think this document is ready for publication.
>
>
>
> Best,
>
> Tommy (as co-chair)
> _______________________________________________
> ippm mailing list
> [email-masked]
> https://www.ietf.org/mailman/listinfo/ippm
>

---

From: Barak Gafni <[email-masked]>
Date: Tue, 21 Jan 2020 23:32:24 +0000

I believe that according to these comments we are converging towards agreement. “SHOULD” doesn’t preclude any implementation hence it keeps the RFC inclusive for devices that can’t align to Bytes. While as we all here for interoperability and simplicity of the designs, recommending the industry to align itself towards standard unit of measurement is desired from a standards body perspective. So I believe that it is aligned with your comments. As you denote, there are cases where standardization of units make sense, while it doesn’t hurt other cases.
In addition, from a personal observation, the market includes many kinds of ASICs implemented as part of switching and routing systems. Some ASIC vendors are reporting their ASIC “cell” sizes while some not, some system/NOS vendors report the ASIC they are using and some not. It may be hard in many environments to query the devices in order to know this data, and then process according. The suggested direction simplifies all this operational aspects when possible. And in the meanwhile, as stated above, doesn’t preclude current implementations.

Thanks,
Barak

From: Mickey Spiegel <[email-masked]>
Sent: Tuesday, January 21, 2020 2:30 PM
To: Barak Gafni <[email-masked]>
Cc: Frank Brockners (fbrockne) <[email-masked]>; Tommy Pauly <tpauly=[email-masked]>; IETF IPPM WG <[email-masked]>
Subject: Re: [ippm] Working Group Last Call: draft-ietf-ippm-ioam-data

On Tue, Jan 21, 2020 at 11:09 AM Barak Gafni <[email-masked]<mailto:[email-masked]>> wrote:
PSB under [BG]

Thanks,
Barak

From: Frank Brockners (fbrockne) <[email-masked]<mailto:[email-masked]>>
Sent: Tuesday, January 21, 2020 5:54 AM
To: Barak Gafni <[email-masked]<mailto:[email-masked]>>; Tommy Pauly <tpauly=[email-masked]<mailto:[email-masked]>>; IETF IPPM WG <[email-masked]<mailto:[email-masked]>>
Subject: RE: [ippm] Working Group Last Call: draft-ietf-ippm-ioam-data

Barak,

Hi,

  1.  Queue depth format – In order to align the standardization of units across the route, as done for time measurements for example, I would suggest to direct implementations towards reporting Bytes, rather than keeping it open for interpretation by the implementors.
…FB: As you know, the original rationale used throughout IOAM is to avoid forcing the data producer (e.g. the forwarding ASIC) to translate a counter into a specific unit, which could be costly. One would rather do the transformation on the receiver of the data than on the sender of the data. At this point you’d probably argue that for time, we also specify the format/unit – though for time, we have 3 options available, and one of which is expected to be supported by the source natively, hence the “exception”.
[BG] Agree, forcing this reporting method may be too much for some implementation that we don’t want to prevent from supporting IOAM. Hence, my suggestion is to use SHOULD, which can drive better interoperability going forward without prevent current implementations from being compatible.

[MS] There are several possible use cases for IOAM. Much of the initial work has been motivated by flow monitoring use cases, where switches and routers embed or export metadata relating to packets in a flow, at line rate for massively scaled data centers. At the receiver, typically the exported metadata is processed and analyzed in software. In such a scenario, in order to enable IOAM to work across as wide a range of devices as possible, it makes sense to use the design philosophy that Frank stated above: Allow the data producer to generate metadata in whatever units it wants, and normalize at the receiver where the exported metadata is processed and analyzed. For such use cases, I believe that standardization of units should not be a goal, not even as a recommendation.

[MS] There may be other use cases where standardization of units makes sense. Rather than constrain all use cases, my suggestion would be to define recommendations specific to such use cases, perhaps using the concept of profiles that Tal has proposed.

  1.  Additional measurements to consider – amount of Bytes transmitted by a port, rate of transmission of a port.

[MS] I support the addition of the amount of bytes transmitted by a port. However, I wonder whether this should be the amount of bytes transmitted on the entire port, or the amount of bytes transmitted using a specific queue on the port?

Mickey

  1.
…FB: The data fields currently defined in the draft are considered a “baseline” – i.e. a set that everyone agrees with. We all know there have been proposals for other data fields in the past – but those proposals never got broader momentum. One can always leverage the opaque state snapshot to carry deployment specific data. So in order to add those two additional data fields to the draft at this stage, we’d need to get a good understanding of how common the need for the two additional data fields is. Who else would see a need? What would be the use-cases?
[BG] Would be great to have the list further comment on that

Thanks, Frank

Thanks,
Barak

Hello IPPM,

Happy New Year! This email starts a working group last call for "Data Fields for In-situ OAM" (https://datatracker.ietf.org/doc/draft-ietf-ippm-ioam-data/<https://eur03.safelinks.protection.outlook.com/?url=https%3A%2F%2Fdatatracker.ietf.org%2Fdoc%2Fdraft-ietf-ippm-ioam-data%2F&data=02%7C01%7Cgbarak%40mellanox.com%7C978b1b54026b488140a808d79ec17cfe%7Ca652971c7d2e4d9ba6a4d149256f461b%7C0%7C0%7C637152426176952430&sdata=YM2cIQkJ7xtvqjq3vBdt%2BYW%2BU8cba2jtkjmzsDWw1QI%3D&reserved=0>).

The last call will end on Tuesday, January 21... Please reply to [email-masked]<mailto:[email-masked]> with your reviews and comments. Please indicate whether you think this document is ready for publication.

Best,
Tommy (as co-chair)
_______________________________________________
ippm mailing list
[email-masked]<mailto:[email-masked]>
https://www.ietf.org/mailman/listinfo/ippm<https://eur03.safelinks.protection.outlook.com/?url=https%3A%2F%2Fwww.ietf.org%2Fmailman%2Flistinfo%2Fippm&data=02%7C01%7Cgbarak%40mellanox.com%7C978b1b54026b488140a808d79ec17cfe%7Ca652971c7d2e4d9ba6a4d149256f461b%7C0%7C0%7C637152426176952430&sdata=S5HIOUimx8%2BSpw39LZnJWLE9gRO1rfazq%2F4WOHosQy8%3D&reserved=0>

---

From: Greg Mirsky <[email-masked]>
Date: Tue, 21 Jan 2020 15:56:49 -0800

Dear All,
please consider my comments to draft-ietf-ippm-ioam-data as part of IPPM WG
Last Call:

   - In Abstract and Section 3, Segment Routing is characterized, among
   other networking technologies, as a transport. RFC 8402 defines Segment
   Routing as a method to leverage the source routing paradigm. Where else
   could the Segment Routing be defined as transport?
   - Also, "native IPv6" is mentioned in the Abstract as transport. Could
   it be just a reference to IPv6? What is the significance of singling out
   "native IPv6"?
   - Introduction section explains the "in-situ" term as

   The term "in-situ" refers to the fact that the OAM
   data is added to the data packets rather than is being sent within
   packets specifically dedicated to OAM.

But with the introduction of Loopback and Direct Export as iOAM behaviors,
iOAM data are not added to the data packet. And how these new iOAM
functions affect the definition of IOAM control points in Section 3?

   - I-D.lapukhov-dataplane-probe referenced as describing "recent active
   probing mechanisms". That could have been the case in 2016 when the draft
   was last time updated. But has expired and has not been discussed or worked
   for 3.5 years.
   - In regard to the control over iOAM, text in Section 3 states "It
   SHOULD be possible to enable IOAM on a selected set of traffic ..." Does
   that mean that not providing the selective enabling of iOAM is an
   acceptable option?
   - Further in Section 3, the requirement for encapsulation independence
   is using SHOULD "Data formats for IOAM SHOULD be defined in a
   transport-independent manner." What are the examples of exceptions?
   - section 4.4 introduces the notion of an "iOAM capable node". Which of
   iOAM objects are expected to be supported by the iOAM capable node? If a
   node doesn't support a sub-set of iOAM data types, would it still be
   considered as iOAM capable?
   - Section 4.4.1 I'm having a hard time figuring how the presence
   of Opaque State Snapshot space can be deduced. The explanation in NodeLen
   is not clear to me.
   - The definitions of Bit 0 and Bit 8 of IOAM-Trace-Type differ only in
   part of the latter specifying that the format is wide. What is the length
   on data identified by Bit 0?
   - Hop_limit in Section 4.4.2 is defined as

        This is copied from the lower
         layer, e.g., TTL value in IPv4 header or hop limit field from
         IPv6 header of the packet when the packet is ready for
         transmission.
What is the value of Hop_Limit field if the lower level does not include
TTL/Hop Limit field?

   - Four octets are allocated for timestamp subseconds in Section 4.4.2.
   Both PTP and NTP have formats that can provide a higher resolution of a
   timestamp. That likely to be useful where a higher resolution of time
   measurement, e.g. delay variation is needed (for example, URLLC
   applications of 5G, TSN or DetNet). Can iOAM data be extended in the future
   to support a different length of the subsecond field?
   - What was the rationale to choose nanoseconds as the unit for transit
   delay?
   - How future proof is allocating four octets to express the length of an
   egress queue (queue depth)? What should be the value if an iOAM node cannot
   write the local value in four octets?
   - Similar to the questions above but in regard to the buffer occupancy
   field.
   - Checksum Complement may be practical but it raises serious security
   issues. I couldn't find these being identified and discussed in the
   document.
   - Section 5 defers the specification of the particular timestamp format
   used in the given iOAM Namespace to the management plane. Does that mean
   that all nodes in the given iOAM Domain must support the same format? In
   other words, timestamps collected in the domain required to be in the
   homogeneous format - either all in NTP, all in PTP, or all in POSIX. That
   looks like unnecessary complexity and limitation and may cause lower
   accuracy of time measurement as an iOAM node may be required to transform
   from its native time format into the format chosen by the management plane.

Regards,
Greg

On Tue, Jan 7, 2020 at 8:47 AM Tommy Pauly <tpauly=
[email-masked]> wrote:

> https://datatracker.ietf.org/doc/draft-ietf-ippm-ioam-data/).
>
> The last call will end on *Tuesday, January 21*. Please reply to

---

From: Mickey Spiegel <[email-masked]>
Date: Tue, 21 Jan 2020 17:52:48 -0800

I believe this draft is ready for publication, after addressing my comments
below.

Comments:

Section 4.6 IOAM Edge-to-Edge Option-Type

In the trace option, the timestamp specifies “the time at which the packet
was received by the node.”

In the edge-to-edge option, it specifies a timestamp “for the transmission
of the frame”.

Why are these different?

Wouldn’t it make more sense to use the time when the packet was received,
in order to determine the overall time that the packet spends in the IOAM
domain?

Section 4.4.1 Pre-allocated and Incremental Trace Option-Types

Text along the following lines should be added somewhere in this section:

An IOAM transit node (that is not an IOAM encapsulating node or IOAM
decapsulating node) MUST NOT modify any of the fields in the fixed size
“trace option header”, other than “flags” and “RemainingLen”, i.e. an IOAM
transit node MUST NOT modify the Namespace-ID, NodeLen, IOAM-Trace-Type, or
Reserved fields.

The “Reserved” field in Section 4.4.1 just says “Must be zero”. That should
be expanded upon:

An IOAM encapsulating node MUST set the value to zero upon transmission.
IOAM transit nodes must ignore the received value.

The “Node data List [n]” description is a little confusing, particularly
the first sentence.

Aside from the first sentence, the rest of this text, as well as the
“RemainingLen” definition, imply that a “node data element” (i.e. list
item) contains all of the IOAM-Data-Fields that a particular node adds to
the trace option.

This implies that the IOAM-Trace-Type bits (collectively) determine which
data fields are included in each node data element.

Any reference to the “n-th node data in the node data list” is misleading
at best, more likely incorrect. That sentence should just be removed.

The last sentence before the “IOAM-Trace-Type” bit definitions could be
moved into the “Node data List [n]” definition instead:

The order of packing the data fields in each node data element follows the
bit order of the IOAM-Trace-Type field

Section 7.4 IOAM Trace-Flags Registry

This section should specify that Bits 1 through 3 are available for
assignment.
A process needs to be selected for assignment of these values, probably RFC
Required.

Section 8 Security Considerations

There is text present about "immediate export" mode.
This should probably be moved to the DEX draft.

Section 9 Acknowledgements

There is text present about "immediate export" mode that should be removed.

Mickey Spiegel
Barefoot Networks, an Intel company

---

From: Greg Mirsky <[email-masked]>
Date: Tue, 21 Jan 2020 22:51:49 -0800

Dear All,
I'd like to add a note on the scope of the Security Considerations section.
While it discusses the potential use of iOAM as the new DDOS attack vector,
I didn't find a discussion of the protection of integrity and
confidentiality of collected data. As I understand, please correct me if
I've missed that in other sections of the document, all data is collected
and transported in the clear text, no provisions for authentication or
encryption except for Proof-of-Transit. I think that in addition to
stressing the importance of protecting the integrity of clock
synchronization and of the management plane, possible ways of protecting
integrity and, possibly, the confidentiality of collected data must be
discussed in the document.

Regards,
Greg

On Tue, Jan 21, 2020 at 3:56 PM Greg Mirsky <[email-masked]> wrote:

> Dear All,
> please consider my comments to draft-ietf-ippm-ioam-data as part of IPPM
> WG Last Call:
>
>    - section 4.4 introduces the notion of an "iOAM capable node". Which
>    of iOAM objects are expected to be supported by the iOAM capable node? If a
>    - How future proof is allocating four octets to express the length of
>    an egress queue (queue depth)? What should be the value if an iOAM node
>    cannot write the local value in four octets?
>    - Section 5 defers the specification of the particular timestamp
>    format used in the given iOAM Namespace to the management plane. Does that
>    mean that all nodes in the given iOAM Domain must support the same format?
>    In other words, timestamps collected in the domain required to be in the
>> [email-masked] with your reviews and comments. Please indicate whether
>> you think this document is ready for publication.
>>
>> Best,
>> Tommy (as co-chair)

---

From: Tal Mizrahi <[email-masked]>
Date: Wed, 22 Jan 2020 10:32:54 +0200

Hi Greg,

Thanks for this comment.

In the context of "confidentiality", since we are talking about OAM
information, the correct context here is reconnaissance. Recon and its
implications are indeed discussed in the Security Considerations section on
the third paragraph.

Regarding integrity protection, the first paragraph of the Security
Considerations section discusses the consequences of compromising the
integrity of IOAM.

In my opinion the current Security Considerations section covers the main
threats related to IOAM, and also explains the rationale of not including
explicit security mechanisms (the last paragraph of the section).

You make a good point that we did not explicitly use the terms
"confidentiality" and "integrity", and I will suggest new text that
explicitly refers to these terms in the relevant context of the section.

Cheers,
Tal.

On Wed, Jan 22, 2020 at 8:52 AM Greg Mirsky <[email-masked]> wrote:

> Dear All,
> I'd like to add a note on the scope of the Security Considerations
> section. While it discusses the potential use of iOAM as the new DDOS
> attack vector, I didn't find a discussion of the protection of integrity
> and confidentiality of collected data. As I understand, please correct me
> if I've missed that in other sections of the document, all data is
> collected and transported in the clear text, no provisions for
> authentication or encryption except for Proof-of-Transit. I think that in
> addition to stressing the importance of protecting the integrity of clock
>>    - Also, "native IPv6" is mentioned in the Abstract as transport.
>>    Could it be just a reference to IPv6? What is the significance of singling
>>    out "native IPv6"?
>> But with the introduction of Loopback and Direct Export as iOAM
>> behaviors, iOAM data are not added to the data packet. And how these new
>> iOAM functions affect the definition of IOAM control points in Section 3?
>>
>>
>>    - I-D.lapukhov-dataplane-probe referenced as describing "recent
>>    active probing mechanisms". That could have been the case in 2016 when the
>>    draft was last time updated. But has expired and has not been discussed or
>>    worked for 3.5 years.
>>    - Further in Section 3, the requirement for encapsulation
>>    independence is using SHOULD "Data formats for IOAM SHOULD be defined in a
>>    - The definitions of Bit 0 and Bit 8 of IOAM-Trace-Type differ only
>>    in part of the latter specifying that the format is wide. What is the
>>    length on data identified by Bit 0?
>>    - Four octets are allocated for timestamp subseconds in Section
>>    4.4.2. Both PTP and NTP have formats that can provide a higher resolution
>>    of a timestamp. That likely to be useful where a higher resolution of time
>>    - What was the rationale to choose nanoseconds as the unit for
>>    transit delay?
>>    - Similar to the questions above but in regard to the buffer
>>    occupancy field.
>> [email-masked] <40apple.com@dmarc..ietf.org>> wrote:
>>
>>> Hello IPPM,
>>>

---

From: Tal Mizrahi <[email-masked]>
Date: Wed, 22 Jan 2020 10:39:56 +0200

Hi Greg,

Regarding your comment: "Checksum Complement may be practical but it raises
serious security issues. I couldn't find these being identified and
discussed in the document."

I am not sure I understand the serious security issues.
In my opinion the Checksum Complement does  not affect security in any way.
The following text regarding the Checksum Complement from RFC 7820
summarizes this point:

   It is important to emphasize that the scheme described in this
   document does not increase the protocol's vulnerability to MITM
   attacks; a MITM attacker who maliciously modifies a packet and its
   Checksum Complement is logically equivalent to a MITM attacker who
   modifies a packet and its UDP Checksum field.

Cheers,
Tal.

---

From: Greg Mirsky <[email-masked]>
Date: Sat, 25 Jan 2020 16:30:01 -0800

Hi Tal,
thank you for your quick response to my comments. Please find my follow-up
notes in-line tagged GIM>>.

Best regards,
Greg

> implications are indeed discussed in the Security Considerations section on
GIM>> I find the text that you've referring too terse:
   The data elements of IOAM can be used for network reconnaissance,
   allowing attackers to collect information about network paths,
   performance, queue states, buffer occupancy and other information.
True, you name the problem but I don't see any discussion of how to
mitigate the risk of an intruder obtaining very sensitive information. What
mechanisms can be used to protect the confidentiality of data?

Also, I've noticed the text that, I think, should not be in the document in
WG Last Call phase:
   Note that in case IOAM is used in "immediate export" mode (reference
   to be added in a future revision) ...
This is a simple editorial change but I'd appreciate it being done.

And as we discuss confidentiality, I believe that
I-D.ietf-sfc-proof-of-transit must be in the normative list since the
security measures and mechanisms discussed in that draft are applicable to
this specification as well.

> Considerations section discusses the consequences of compromising the
GIM>> I am not sure that I can interpret the first paragraph in the same
manner as you. (As a side note, since RFC 7276 is used as the source of
information, I think it should be in the normative list of references). My
understanding of the first paragraph is that it explains that iOAM may be
used to produce a false-negative and/or false-positive view of the network
state. But I don't find any discussion on how to protect, how to make it
more difficult for an attacker to manipulate with or alter iOAM data.

> explicit security mechanisms (the last paragraph of the section).
GIM>> On this I do agree. Security Considerations section names threats.
But I believe that it as well should include recommendations, options of
mitigating these risks, risk introduced specifically by iOAM transporting
information in data packets.

>>>    - How future proof is allocating four octets to express the length
>>>    of an egress queue (queue depth)? What should be the value if an iOAM node
>>>    - Checksum Complement may be practical but it raises serious
>>>    security issues. I couldn't find these being identified and discussed in
>>>    the document.

---

From: Tal Mizrahi <[email-masked]>
Date: Mon, 27 Jan 2020 10:52:26 +0200

Hi Greg,

Many thanks for the comments.
Your point is well taken regarding the term "immediate export", and
regarding clarifying the connection to integrity and confidentiality.
There is a fundamental assumption in this draft that since IOAM is deployed
in confined administrative domains, it significantly limits the scope of
all the threats that are described in this section. Therefore this draft
does not define any specific security mechanisms. This was a basic
assumption throughout the process, and therefore the security
considerations does not define any security mechanisms. However, I added
some text that further clarifies this point.

The following pull request includes my suggested changes to the Security
Considerations following your comments:
https://github.com/inband-oam/ietf/pull/146

Cheers,
Tal.

On Sun, Jan 26, 2020 at 2:30 AM Greg Mirsky <[email-masked]> wrote:

> Also, I've noticed the text that, I think, should not be in the document
> in WG Last Call phase:
>>>> please consider my comments to draft-ietf-ippm-ioam-data as part of
>>>> IPPM WG Last Call:
>>>>
>>>>    - In Abstract and Section 3, Segment Routing is characterized,
>>>>    among other networking technologies, as a transport. RFC 8402 defines
>>>>    Segment Routing as a method to leverage the source routing paradigm. Where
>>>>    else could the Segment Routing be defined as transport?
>>>>    - section 4.4 introduces the notion of an "iOAM capable node".
>>>>    Which of iOAM objects are expected to be supported by the iOAM capable
>>>>    node? If a node doesn't support a sub-set of iOAM data types, would it
>>>>    still be considered as iOAM capable?
>>>> What is the value of Hop_Limit field if the lower level does not
>>>> include TTL/Hop Limit field?
>>>>
>>>>

---

From: Tommy Pauly <[email-masked]>
Date: Tue, 28 Jan 2020 07:55:25 -0800

[Resending as this email is missing from the mail archive]

Hi IPPM,

Thanks for the feedback on this draft-ietf-ippm-ioam-data!

Based on the reviews, we would like to see a new version of the document posted, so that the group can take a look at those changes and see if we think we're ready after that.

Authors, please post a new version when ready.

Best,
Tommy

> On Jan 7, 2020, at 8:47 AM, Tommy Pauly <tpauly=[email-masked] <mailto:tpauly=[email-masked]>> wrote:
> 
> Hello IPPM,
> 
> Happy New Year! This email starts a working group last call for "Data Fields for In-situ OAM" (https://datatracker.ietf.org/doc/draft-ietf-ippm-ioam-data/ <https://datatracker.ietf.org/doc/draft-ietf-ippm-ioam-data/>).
> 
> The last call will end on Tuesday, January 21. Please reply to [email-masked] <mailto:[email-masked]> with your reviews and comments. Please indicate whether you think this document is ready for publication.
> 
> Best,
> Tommy (as co-chair)
> [email-masked] <mailto:[email-masked]>

---

From: Greg Mirsky <[email-masked]>
Date: Tue, 28 Jan 2020 17:08:56 -0800

Hi Tal,
thank you for the clarifying details. I believe it is highly risky to leave
outside the document the discussion of what security mechanisms, measures
are required from an iOAM domain to be considered as a "confined
administrative domain". I strongly encourage adding it to the document as
one of the sections. Then, these listed mechanisms and principles can be
referenced from the Security Considerations section to explain why iOAM
itself does not require the additional measures to protect data integrity
and/or confidentiality.

Regards,
Greg

On Mon, Jan 27, 2020 at 12:52 AM Tal Mizrahi <[email-masked]>
wrote:

> There is a fundamental assumption in this draft that since IOAM is
> deployed in confined administrative domains, it significantly limits the
> scope of all the threats that are described in this section. Therefore this
> draft does not define any specific security mechanisms. This was a basic
>> thank you for your quick response to my comments. Please find my
>> follow-up notes in-line tagged GIM>>.
>>
>> Best regards,
>> Greg
>>
>>> In my opinion the current Security Considerations section covers the
>>> main threats related to IOAM, and also explains the rationale of not
>>> including explicit security mechanisms (the last paragraph of the section).
>>>
>>>>>    - The definitions of Bit 0 and Bit 8 of IOAM-Trace-Type differ
>>>>>    only in part of the latter specifying that the format is wide. What is the
>>>>>> [email-masked] with your reviews and comments. Please indicate
>>>>>> whether you think this document is ready for publication.
>>>>>>
>>>>>> Best,
>>>>>> Tommy (as co-chair)