Subject: [pm-dir] Performance directorate review of draft-ietf-alto-deployments

From: Vinayak Hegde <[email-masked]>
Date: Sat, 7 Sep 2013 21:52:50 +0530

Hi,

There was a performance review either requested by you or was triggered by
Benoit's script. I am reviewing this from a performance metrics point of
view.

I looked at the draft (
https://datatracker.ietf.org/doc/draft-ietf-alto-deployments/?include_text=1)
These comments are only on the Section 6.3 Monitoring ALTO

Comments on Section 6.3
The authors should define some recommended metrics here (possibly IPPM
Type-P metrics) if not recommended metric here. This section can be
substantially expanded.

Comments on section 6.3.1
There are several metrics defined here but they can be more clearly defined
by following the formats specified in RFC 6390. Also useful will some
use-cases to explain why these metrics make sense and what needs to be used
in which scenario.

As regards to the network hop-count to be used indirectly for latency, it
might be better to define a different metric for latency as the average
network loads and buffers on each device can be radically different. For
example a hybrid network that contains  a backbone network, last mile
access network and a mobile network (which is not unusual in many ISPs
today). The delay for transit (as well as packet loss) internally from one
to other might be drastically different.

Also it might be useful to use passive monitoring in the case of some
metrics such as application download rate.

Section 6.3.3
The example given here can be expanded to give a more clearer picture of
which are the active monitoring agents and which are the passive
components. The metrics used for monitoring need to standardized. Also you
might want to add some recommendation on the data interchange format and
method of delivery. This is especially important as the interacting
entities (ISP and Application Owner) are present at different layers and
have different priorities.This can also be fixed in section 6.3.1 above.

Thanks
Vinayak

---

From: "Scharf, Michael (Michael)" <[email-masked]>
Date: Fri, 13 Sep 2013 13:20:20 +0000

Hi Vinayak,

Thanks a lot for the feedback! We are currently revising this document, and I agree that Section 6.3 needs substantial rewording. More comments inline [MS]...

Thanks again!

Michael

________________________________
From: Vinayak Hegde [mailto:[email-masked]]
Sent: Saturday, September 07, 2013 6:23 PM
To: [email-masked]; [email-masked]; [email-masked]; Scharf, Michael (Michael)
Cc: [email-masked]
Subject: Performance directorate review of draft-ietf-alto-deployments

Hi,

There was a performance review either requested by you or was triggered by Benoit's script. I am reviewing this from a performance metrics point of view.

I looked at the draft (https://datatracker.ietf.org/doc/draft-ietf-alto-deployments/?include_text=1) These comments are only on the Section 6.3 Monitoring ALTO

Comments on Section 6.3
The authors should define some recommended metrics here (possibly IPPM Type-P metrics) if not recommended metric here. This section can be substantially expanded.

[MS] I am not sure if this document should really formally define metrics (the current wording implies that). The more important aspect seems to me the overall effects that an operator may want to look at when deploying ALTO. This significantly differs between different use cases of ALTO. For instance, if ALTO is used for traffic optimization inside one AS, monitoring a metric that describes inter-domain traffic may not be required. Therefore, the relevant metrics will depend on the use case, as also noted in your review. An alternative document structure would have some discussion of monitoring per use case, but this would imply a significant change of the table of content of the document. We currently look into this, but this requires feedback from the ALTO WG, too.
Comments on section 6.3.1
There are several metrics defined here but they can be more clearly defined by following the formats specified in RFC 6390. Also useful will some use-cases to explain why these metrics make sense and what needs to be used in which scenario.

[MS] Agreed. As I said, one potential solution is to move this discussion to the descriptions of the use cases (Section 4, 5, etc.), maybe with one "monitoring" subsection per use-case.

As regards to the network hop-count to be used indirectly for latency, it might be better to define a different metric for latency as the average network loads and buffers on each device can be radically different. For example a hybrid network that contains  a backbone network, last mile access network and a mobile network (which is not unusual in many ISPs today). The delay for transit (as well as packet loss) internally from one to other might be drastically different.

[MS] I agree that hop-count and latency may not correlate. Also, measuring hop-count can be difficult in some cases. If the Internet traffic is carried over an MPLS/IP aggregation/core network, the number of routers that are traversed by the traffic can be pretty different to the number of IP hops. Reducing the IP hops does not necessarily imply a more efficient routing. My current thinking is that the document should more focus on "traffic locality" than path monitoring metrics.
Also it might be useful to use passive monitoring in the case of some metrics such as application download rate.

[MS] Yes, the document probably should reference IPFIX, LMAP, etc.

Section 6.3.3
The example given here can be expanded to give a more clearer picture of which are the active monitoring agents and which are the passive components. The metrics used for monitoring need to standardized. Also you might want to add some recommendation on the data interchange format and method of delivery. This is especially important as the interacting entities (ISP and Application Owner) are present at different layers and have different priorities.This can also be fixed in section 6.3.1 above.

[MS] Yes, the text and Figure 19 describes the scenario at a very high-level and, indeed, it is not clear. Both the question what somebody can measure, and whether this is done actively or passively, depends on the use case and who that entity actually is. But I don't believe that this document can provide comprehensive guidance on data exchange formats and methods of delivery, because inter-organization communication raises a lot of non-technical issues, and even standardization of new protocols might be needed. That seems outside the scope of this document specifically, and ALTO in general. This seems more related to LMAP.

 Thanks
Vinayak

---

From: Vinayak Hegde <[email-masked]>
Date: Mon, 23 Sep 2013 15:03:09 +0530

On Fri, Sep 13, 2013 at 6:50 PM, Scharf, Michael (Michael) <
[email-masked]> wrote:

> **
> [snipped]
You are welcome. Comments inline.

> ------------------------------
> *From:* Vinayak Hegde [mailto:[email-masked]]
> *Sent:* Saturday, September 07, 2013 6:23 PM
> *To:* [email-masked]; [email-masked];
> [email-masked]; Scharf, Michael (Michael)
> *Cc:* [email-masked]
> *Subject:* Performance directorate review of draft-ietf-alto-deployments
>
>    Hi,
>
>  There was a performance review either requested by you or was triggered
> by Benoit's script. I am reviewing this from a performance metrics point of
> view.
>
>  I looked at the draft (
> [MS] I am not sure if this document should really formally define metrics
> (the current wording implies that). The more important aspect seems to
> me the overall effects that an operator may want to look at when deploying
> ALTO. This significantly differs between different use cases of ALTO.
> For instance, if ALTO is used for traffic optimization inside
> one AS, monitoring a metric that describes inter-domain traffic may not be
> required. Therefore, the relevant metrics will depend on the use case, as
> also noted in your review. An alternative document structure would have
> some discussion of monitoring per use case, but this would imply a
> significant change of the table of content of the document. We currently
> look into this, but this requires feedback from the ALTO WG, too.
>
> Yeah a couple of use-cases would really help though it is bit more work.
It would make the document more complete and usable for implementers.

>  There are several metrics defined here but they can be more clearly
> defined by following the formats specified in RFC 6390. Also useful will
> some use-cases to explain why these metrics make sense and what needs to be
> used in which scenario.
>
> [MS] Agreed. As I said, one potential solution is to move this discussion
> to the descriptions of the use cases (Section 4, 5, etc.), maybe with one
> "monitoring" subsection per use-case.
>
>
Yup. Agree. Makes sense.

> [MS] I agree that hop-count and latency may not correlate. Also, measuring
> hop-count can be difficult in some cases. If the Internet traffic is
> carried over an MPLS/IP aggregation/core network, the number of routers
> that are traversed by the traffic can be pretty different to the number of
> IP hops. Reducing the IP hops does not necessarily imply a more efficient
> routing. My current thinking is that the document should more focus on
> "traffic locality" than path monitoring metrics.
>
>
Agree. Makes sense. Can you reword the mentioned paragraph ?

> [MS] Yes, the text and Figure 19 describes the scenario at a very
> high-level and, indeed, it is not clear. Both the question what somebody
> can measure, and whether this is done actively or passively, depends on the
> use case and who that entity actually is. But I don't believe that
> this document can provide comprehensive guidance on data exchange formats
> and methods of delivery, because inter-organization communication raises a
> lot of non-technical issues, and even standardization of new protocols
> might be needed. That seems outside the scope of this document
> specifically, and ALTO in general. This seems more related to LMAP.
>
>
The document need not be comprehensive. That was not the intention of the
suggestion. But non-normative language about possible measures / metrics
could be useful for guidance. Data interchange formats such as JSON could
be used for easy parsing instead of binary serialisation.

-- Vinayak