Subject: [lmap] FW: AD review of draft-ietf-lmap-use-cases-03

From: <[email-masked]>
Date: Fri, 12 Sep 2014 15:53:28 +0100

cfi - in case there needs to be any discussion during Monday's interim, here are the proposed resolutions to Benoit's comments.
best wishes,
phil

________________________________
From: [email-masked] [[email-masked]]
Sent: 03 September 2014 14:20
To: [email-masked]; [email-masked]
Subject: RE: [lmap] AD review of draft-ietf-lmap-use-cases-03

Benoit,
Thank-you very much for your review. Sorry for the slow response - crossed wires amongst the authors

I’ve proposed resolutions to all your comments below – except for one that I don’t understand
<<- Section 3.1

   Operators want to understand the quality of experience (QoE) of their

   broadband customers.
There is a QoE reference in RFC 5481

[phil] This was clarified:-
The QoE reference is actually https://tools.ietf.org/html/rfc6390#section-2.4

[to try and close the issues more rapidly, am emailing the use cases authors rather than lmap list.]

Thanks!
phil

From: lmap [mailto:[email-masked]] On Behalf Of Benoit Claise
Sent: 24 July 2014 15:30
To: [email-masked]
Subject: [lmap] AD review of draft-ietf-lmap-use-cases-03

Dear all,

Here is my AD review

- OLD

Abstract

   Measuring broadband performance on a large scale is important for

   network diagnostics by providers and users, as well as for public

   policy.  Understanding the various scenarios and users of measuring

   broadband performance is essential to development of the framework,

   information model and protocol. This document details two use cases

   that can assist to developing that framework.  The details of the

   measurement metrics themselves are beyond the scope of this document.
NEW:

Abstract

   Measuring broadband performance on a large scale is important for

   network diagnostics by providers and users, as well as for public

   policy.  Understanding the various scenarios and users of measuring

   broadband performance is essential to development of the Large-scale Measurement

   of Broadband Performance (LMAP) framework,

   information model and protocol. This document details two use cases

   that can assist to developing that framework.  The details of the

   measurement metrics themselves are beyond the scope of this document.

Yes – agree.

-  Section 2.1

   An ISP, or indeed another network operator, needs to understand the

   performance of their networks, the performance of the suppliers

   (downstream and upstream networks), the performance of services, and

   the impact that such performance has on the experience of their

   customers.
"indeed another network operator"?

NEW:

   A network operator needs to understand the

   performance of their networks, the performance of the suppliers

   (downstream and upstream networks), the performance of services, and

   the impact that such performance has on the experience of their

   customers.

- Section 2.1

      Identifying, isolating and fixing problems in the network,

      services or with CPE and end user equipment.

I can't parse the sentence

NEW

      Identifying, isolating and fixing problems, which may be in the network,

      with the service provider, or in the end user equipment.

-  Section 2.1

This sentence speaks about "services"

      o Identifying, isolating and fixing problems in the network,

      services or with CPE and end user equipment.

Section 3.2 speaks about new services.

So, this paragraph also includes "services":

      o Understanding the impact and operation of new devices and

      technology. As a new product is deployed, or a new technology

      introduced into the network, it is essential that its operation

      and impact on other services is measured. This also helps to

      quantify the advantage that the new technology is bringing and

      support the business case for larger roll-out.

NEW:

      o Understanding the impact and operation of new devices,

      technology, or services. As a new product is deployed, or a new technology

      introduced into the network, it is essential that its operation

      and impact on other services is measured. This also helps to

      quantify the advantage that the new technology is bringing and

      support the business case for larger roll-out.

I think ‘services’ here & in S3.2 is mainly being used in the sense of ‘network services’, which is confusing.
NEW

      o Understanding the impact and operation of new devices and

      technology. As a new product is deployed, or a new technology

      introduced into the network, it is essential that its operation

      and its impact is measured. This also helps to

      quantify the advantage that the new technology is bringing and

      support the business case for larger roll-out.

and in S3.2, first sentence
OLD
Another type of measurement is to test new capabilities and services
   before they are rolled out.

NEW
Another type of measurement is to test new capabilities
   before they are rolled out.

- I'm confused by:

 2<http://tools.ietf.org/html/draft-ietf-lmap-use-cases-03#section-2>  Use Cases . . . . . . . . . . . . . . . . . . . . . . . . . . .  3<http://tools.ietf.org/html/draft-ietf-lmap-use-cases-03#page-3>

     2.1<http://tools.ietf.org/html/draft-ietf-lmap-use-cases-03#section-2.1> Internet Service Provider (ISP) Use Case . . . . . . . . . .  3<http://tools.ietf.org/html/draft-ietf-lmap-use-cases-03#page-3>

     2.2<http://tools.ietf.org/html/draft-ietf-lmap-use-cases-03#section-2.2> Regulators . . . . . . . . . . . . . . . . . . . . . . . . .  4<http://tools.ietf.org/html/draft-ietf-lmap-use-cases-03#page-4>

     2.3<http://tools.ietf.org/html/draft-ietf-lmap-use-cases-03#section-2.3> Implementation options . . . . . . . . . . . . . . . . . . .  5<http://tools.ietf.org/html/draft-ietf-lmap-use-cases-03#page-5>
What is the link between the "Implementation options" subsection and use cases?
Should "implementation options" have its own section?

I agree. Think it fits best as a section near the end.
NEW
   5<http://tools.ietf.org/html/draft-ietf-lmap-use-cases-03#section-5>  Implementation options . . . . . . . . . . . . . . . . . . .  12<http://tools.ietf.org/html/draft-ietf-lmap-use-cases-03#page-12>
   6  Conclusions . . . . . . . . . . . . . . . . . . . . . . . . . . 12<http://tools.ietf.org/html/draft-ietf-lmap-use-cases-03#page-12>

-

   Another approach involves implementing the measurement capability as

   a webpage or an "app" that end users are encouraged to download onto

   their mobile phone or computing device. Measurements are triggered by

   the end user, for example the user interface may have a button to

   "test my broadband now". Compared with the previous approach, the

   system is much more loosely controlled, as the panel of end users and

   the schedule of tests are determined by the end users themselves

   rather than the measurement system. It would be easier to get large-

   scale, however it is harder to get comparable benchmarks as the

   measurements are affected by the home network and also the population

   is self-selecting and so potentially biased towards those who think

   they have a problem. This could be alleviated by stimulating

   widespread downloading of the app and careful post-processing of the

   results to reduce biases.
This end-user triggered measurement provides a huge advantage: it reflects the end-user QoE, as opposed to the broadband device QoE (*).  In combination with LMAP measurements in the broadband device, the end-user triggered measurement might prove that the home network is the source of performance degradations.
You might want to add those points either in section "Implementation options" or around the section 3.1  3rd paragraph.
This point is partly covered in last paragraph of section 3.5

(*) except if the end-user triggered measurement is executed directly on the broadband device ... For example, downloading a page, or initiating a test from the web server on the broadband device, as you mentioned:

   There are several other possibilities. For example, as a variant on

   the first approach, the measurement capability could be implemented

   as software embedded in the home gateway, which would make it more

   viable to have the capability on every user line. As a variant on the

   second approach, the end user could initiate measurements in response

   to a request from the measurement system.

NEW

   Another approach involves implementing the measurement capability as

   a webpage or an "app" that end users are encouraged to download onto

   their mobile phone or computing device. Measurements are triggered by

   the end user, for example the user interface may have a button to

   "test my broadband now". One advantage of this approach is that the performance is measured to the end user, rather than to the home gateway, and so includes the home network. Another difference is that the

   system is much more loosely controlled, as the panel of end users and

   the schedule of tests are determined by the end users themselves

   rather than the measurement system. It would be easier to get large-

   scale, however it is harder to get comparable benchmarks as the

   measurements are affected by the home network and also the population

   is self-selecting and so potentially biased towards those who think

   they have a problem. This could be alleviated by stimulating

   widespread downloading of the app and careful post-processing of the

   results to reduce biases.

- In connection with the previous point,  I was confused while reading the document.
I don't see"end-user triggered measurement" in section 2.X next to the ISP and operators use case. fine, but it's still discussed throughout the draft. For example

   Measurements are triggered by

   the end user, for example the user interface may have a button to

   "test my broadband now".

   OR

   As a variant on the

   second approach, the end user could initiate measurements in response

   to a request from the measurement system.

   OR

   The operator really wants to understand the end-to-end service

   experience. However, the home network (Ethernet, WiFi, powerline) is

   highly variable and outside its control. To date, operators (and

   regulators) have instead measured performance from the home gateway.

   However, mobile operators clearly must include the wireless link in

   the measurement.

   OR

   The operator would like the

   end-to-end view of the service, rather than (say) just the access

   portion

So, I've been wondering whether this end-user triggered measurement was in scope or not.
I only found my answer in the conclusion section

   There are

   other use cases that are not the focus of the initial LMAP charter

   (although it is expected that the mechanisms developed would be

   readily applied), for example end users would like to use

   measurements to help identify problems in their home network and to

   monitor the performance of their broadband provider.
You should make it very clear, upfront, that the end-user triggered measurement is not in scope.
Something like section 5.6.1 End-user-controlled measurement system in the framework document

Suggest changing S1 Introduction
OLD
   This document describes two use cases for the Large-scale Measurement
   of Broadband Performance (LMAP), in particular use cases for ISPs and
   regulators.  Although there are many other use cases for large-scale
   measurements systems, the two described here are the consensus
   starting point for defining the system.
NEW
   This document describes two use cases for the Large-scale Measurement

   of Broadband Performance (LMAP). Firstly, to enable network operators to understand the performance of the network and the quality experienced by customers. Secondly, to enable regulators to provide information on the performance of the ISPs in their jurisdiction. There are other use cases that are not the focus of the initial LMAP work, for example end users would like to use measurements to help identify problems in their home network and to monitor the performance of their broadband provider; it is expected that the same mechanisms are applicable.

- Section 3.1

   Operators want to understand the quality of experience (QoE) of their

   broadband customers.
There is a QoE reference in RFC 5481

I don’t understand this comment, sorry. I couldn’t find “QoE” in RFC5481

- Section 3.5

   One example was described in [IETF85-Plenary]. The operator was

   running a measurement panel for reasons discussed in sub use case #1.
Should we all review [IETF85-Plenary] to understand what sub use case # 1 is? ;-)

This meant sub use case #1 in draft-lmap-use-cases. However, I think the current text can be improved.

OLD
   One example was described in [IETF85-Plenary<http://tools.ietf.org/html/draft-ietf-lmap-use-cases-03#ref-IETF85-Plenary>]. The operator was
   running a measurement panel for reasons discussed in sub use case #1.
   It was noticed that the performance of some lines had unexpectedly
   degraded. This led to a detailed (off-line) investigation which
   discovered that a particular home gateway upgrade had caused a
   (mistaken!) drop in line rate.

   Another example is that occasionally some internal network management
   event (like re-routing) can be customer-affecting (of course this is
   unusual). This affects a whole group of customers, for instance those
   on the same DSLAM. Understanding this will help an operator fix the
   fault more rapidly and/or allow the affected customers to be informed
   what's happening and/or request them to re-set their home hub
   (required to cure some conditions). More accurate information enables
   the operator to reassure customers and take more rapid and effective
   action to cure the problem.

   There may also be problems unique to a single user line (e.g. copper
   access) that need to be identified.

NEW

An operator can obtain useful information without measuring the performance on every broadband line. By measuring a subset, the operator identify problems that affect a group of customers. For example, the issue could be at a shared point in the network topology (such as an exchange) or common to a vendor or equipment type ([IETF85-Plenary] describes a case where a particular home gateway upgrade had caused a (mistaken!) drop in line rate). A more extensive deployment of the measurement capability to every broadband line would enable an operator to identify issues unique to a single customer. Overall, large-scale measurements can help an operator help an operator fix the fault more rapidly and/or allow the affected customers to be informed what's happening. More accurate information enables the operator to reassure customers and take more rapid and effective action to cure the problem.

- Section 4.1

   The published information needs to be:

      o  Accurate - the measurement results must be correct and not

      influenced by errors or side effects. The results should be

      reproducible and consistent over time.

      o  Comparable - common metrics should be used across different

      ISPs and service offerings so that measurement results can be

      compared.

      o  Meaningful - the metrics used for measurements need to reflect

      what end users value about their broadband Internet access service

      o  Reliable - the number and distribution of measurement agents,

      and the statistical processing of the raw measurement raw data,

      needs to be appropriate

Not sure if this was discussed, but one goal behind the regulator use case is to strongly suggest the ISPs to use the same metrics in their SLC (Service Level Contracts).
If you ever tried to compare SLC between ISPs, well, go luck.
This would be a nice additional 4.x section

Rather than a new section, I suggest an additional sentence in the 2nd para of S4.1:
NEW
   End users need effective transparency to be able to make informed
   choices throughout the different stages of their relationship with
   ISPs, when selecting Internet access service offers, and when
   considering switching service offer within an ISP or to an
   alternative ISP. Quality information about service offers could
   include speed, delay, and jitter. Regulators can publish such
   information to facilitate end users' choice of service provider and
   offer. It may also encourage ISPs to use the same metrics in their service level contracts, which would further help end users to choose an ISP. Finally, transparency may help content, application, service and device
   providers develop their Internet offerings.

- Pervasive Monitoring, RFC 7528. This RFC should at least be referenced, for example around this sentence:

   The other approach is passive measurements

   on the customer's ordinary traffic; the advantage is that it measures

   what the customer actually does, but it creates extra variability

   (different traffic mixes give different results) and especially it

   raises privacy concerns.

Do you mean RFC6973? (Privacy Considerations for Internet Protocols)
NEW

   The other approach is passive measurements

   on the customer's ordinary traffic; the advantage is that it measures

   what the customer actually does, but it creates extra variability

   (different traffic mixes give different results) and especially it

   raises privacy concerns. [RFC6973] discusses privacy considerations for Internet protocols in general, whilst [framework] discusses them specifically for large-scale measurement systems.

And add to references:

   [RFC6973]  Cooper, A., Tschofenig, H., Aboba, B., Peterson, J.,
              Morris, J., Hansen, M., and R. Smith, "Privacy
              Considerations for Internet Protocols", RFC 6973<http://tools.ietf.org/html/rfc6973>, July
              2013.

Regards, Benoit

--
Nits spotted:
S2.2
Programs /programmes (twice)

S2.2, end of penultimate paragraph is missing a full stop.

S3.5 Large scale measurements can help provide a more nuance view that

==>  Large-scale measurements can help provide a more nuanced view that

S4.3 take away for LMAP purposes are that policy-makers are looking for

==>  take-away for LMAP purposes is that policy-makers are looking for

---

From: Benoit Claise <[email-masked]>
Date: Sun, 14 Sep 2014 18:29:23 +0200

Hi Phil,

I'm happy with the way you addressed my feedback.
Just one point needs correction: the last one.

    Pervasive Monitoring, RFC 7528. This RFC should at least be
    referenced, for example around this sentence:

        raises privacy concerns.

It's actually my fault. Wrong reference! it should be RFC 7258

Regards, Benoit
> cfi - in case there needs to be any discussion during Monday's 
> interim, here are the proposed resolutions to Benoit's comments.
> best wishes,
> phil
> ------------------------------------------------------------------------
> *From:* [email-masked] [[email-masked]]
> *Sent:* 03 September 2014 14:20
> *To:* [email-masked]; [email-masked]
> *Subject:* RE: [lmap] AD review of draft-ietf-lmap-use-cases-03
>
> Benoit,
>
> I've proposed resolutions to all your comments below -- except for one 
> that I don't understand
>
> <<- Section 3.1
>
> The QoE reference is 
> actuallyhttps://tools.ietf.org/html/rfc6390#section-2.4
>
> *From:*lmap [mailto:[email-masked]] *On Behalf Of *Benoit Claise
> *Sent:* 24 July 2014 15:30
> *To:* [email-masked]
> *Subject:* [lmap] AD review of draft-ietf-lmap-use-cases-03
>
> Dear all,
>
> Here is my AD review
>
> - OLD
>
> Abstract
>   
>     broadband performance is essential to development of the_Large-scale Measurement_
> _    of Broadband Performance (LMAP)_  framework,
> Yes -- agree.
> "indeed _another _network operator"?
>
> NEW:
>
>     A network operator needsto understand the
>        Identifying, isolating and fixingproblems, which may be in the network,
>        with the service provider, or in theend user equipment.
>   
>
>
> -  Section 2.1
>
>        technology,_or services._  As a new product is deployed, or a new technology
> I think 'services' here & in S3.2 is mainly being used in the sense of 
> 'network services', which is confusing.
>
> NEW
>
>        andits impact ismeasured. This also helps to
>   2  <http://tools.ietf.org/html/draft-ietf-lmap-use-cases-03#section-2>   Use Cases . . . . . . . . . . . . . . . . . . . . . . . . . . .3  <http://tools.ietf.org/html/draft-ietf-lmap-use-cases-03#page-3>
>       2.1  <http://tools.ietf.org/html/draft-ietf-lmap-use-cases-03#section-2.1>  Internet Service Provider (ISP) Use Case . . . . . . . . . .3  <http://tools.ietf.org/html/draft-ietf-lmap-use-cases-03#page-3>
>       2.2  <http://tools.ietf.org/html/draft-ietf-lmap-use-cases-03#section-2.2>  Regulators . . . . . . . . . . . . . . . . . . . . . . . . .4  <http://tools.ietf.org/html/draft-ietf-lmap-use-cases-03#page-4>
>       2.3  <http://tools.ietf.org/html/draft-ietf-lmap-use-cases-03#section-2.3>  Implementation options . . . . . . . . . . . . . . . . . . .5  <http://tools.ietf.org/html/draft-ietf-lmap-use-cases-03#page-5>
>
> 5 <http://tools.ietf.org/html/draft-ietf-lmap-use-cases-03#section-5> 
> Implementation options . . . . . . . . . . . . . . . . . . . 12 
> <http://tools.ietf.org/html/draft-ietf-lmap-use-cases-03#page-12>
>
>    6  Conclusions . . . . . . . . . . . . . . . . . . . . . . . . . . 
> 12 <http://tools.ietf.org/html/draft-ietf-lmap-use-cases-03#page-12>
>
>
>
> -
>
> This end-user triggered measurement provides a huge advantage: it 
> reflects the end-user QoE, as opposed to the broadband device QoE 
> (*).  In combination with LMAP measurements in the broadband device, 
> the end-user triggered measurement might prove that the home network 
> is the source of performance degradations.
> You might want to add those points either in section "Implementation 
> options" or around the section 3.1 3rd paragraph.
> (*) except if the end-user triggered measurement is executed directly 
> on the broadband device ... For example, downloading a page, or 
> initiating a test from the web server on the broadband device, as you 
> mentioned:
>
>     "test my broadband now".One advantage of this approach is that the performance is measured to the end user, rather than to the home gateway, and so includes the home network. Another difference is thatthe
> I don't see"end-user triggered measurement" in section 2.X next to the 
> ISP and operators use case. fine, but it's still discussed throughout 
> the draft. For example
>
> I don't understand this comment, sorry. I couldn't find "QoE" in RFC5481
>
>
>
> - Section 3.5
>
>    One example was described in [IETF85-Plenary 
> <http://tools.ietf.org/html/draft-ietf-lmap-use-cases-03#ref-IETF85-Plenary>]. 
> The operator was
>
> An operator can obtain useful information without measuring the 
> performance on every broadband line. By measuring a subset, the 
> operator identify problems that affect a group of customers. For 
> example, the issue could be at a shared point in the network topology 
> (such as an exchange) or common to a vendor or equipment type 
> ([IETF85-Plenary] describes a case where a particular home gateway 
> upgrade had caused a (mistaken!) drop in line rate). A more extensive 
> deployment of the measurement capability to every broadband line would 
> enable an operator to identify issues unique to a single customer. 
> Overall, large-scale measurements can help an operator help an 
> operator fix the fault more rapidly and/or allow the affected 
> customers to be informed what's happening. More accurate information 
> enables the operator to reassure customers and take more rapid and 
> effective action to cure the problem.
>
>
> - Section 4.1
>
> Not sure if this was discussed, but one goal behind the regulator use 
> case is to strongly suggest the ISPs to use the same metrics in their 
> SLC (Service Level Contracts).
> Rather than a new section, I suggest an additional sentence in the 
> 2^nd para of S4.1:
>
> NEW
>
>    offer. It may also encourage ISPs to use the same metrics in their 
> service level contracts, which would further help end users to choose 
> an ISP. Finally, transparency may help content, application, service 
> and device
>
> - Pervasive Monitoring, RFC 7528. This RFC should at least be 
> referenced, for example around this sentence:
>
>     raises privacy concerns.[RFC6973] discusses privacy considerations for Internet protocols in general, whilst [framework] discusses them specifically for large-scale measurement systems.
>
> And add to references:
>
>               Considerations for Internet Protocols", RFC 6973 
> <http://tools.ietf.org/html/rfc6973>, July
>
>               2013.
>
> Regards, Benoit
>
> --
>
> Nits spotted:
>
> S2.2
>
> èLarge-scale measurements can help provide a more nuanced view that
>
> ètake-away for LMAP purposes is that policy-makers are looking for
>
>
>
> _______________________________________________
> lmap mailing list
> [email-masked]
> https://www.ietf.org/mailman/listinfo/lmap

---

From: Benoit Claise <[email-masked]>
Date: Mon, 22 Sep 2014 14:21:14 +0200

Hi Phil,

As soon as you post a new version, I'll send this draft to IETF LC.

Regards, Benoit
>> I've proposed resolutions to all your comments below -- except for 
>> one that I don't understand
>>
>> <<- Section 3.1
>>
>> I think 'services' here & in S3.2 is mainly being used in the sense 
>> of 'network services', which is confusing.
>>
>> NEW
>>
>> I don't see"end-user triggered measurement" in section 2.X next to 
>> the ISP and operators use case. fine, but it's still discussed 
>> throughout the draft. For example
>>
>> deployment of the measurement capability to every broadband line 
>> would enable an operator to identify issues unique to a single 
>> customer. Overall, large-scale measurements can help an operator help 
>> an operator fix the fault more rapidly and/or allow the affected