From dromasca@avaya.com Tue Nov 5 07:16:55 2013 Return-Path: X-Original-To: lmap@ietfa.amsl.com Delivered-To: lmap@ietfa.amsl.com Received: from localhost (localhost [127.0.0.1]) by ietfa.amsl.com (Postfix) with ESMTP id C9FE211E81DD for ; Tue, 5 Nov 2013 07:16:55 -0800 (PST) X-Virus-Scanned: amavisd-new at amsl.com X-Spam-Flag: NO X-Spam-Score: -103.321 X-Spam-Level: X-Spam-Status: No, score=-103.321 tagged_above=-999 required=5 tests=[AWL=0.278, BAYES_00=-2.599, RCVD_IN_DNSWL_LOW=-1, USER_IN_WHITELIST=-100] Received: from mail.ietf.org ([12.22.58.30]) by localhost (ietfa.amsl.com [127.0.0.1]) (amavisd-new, port 10024) with ESMTP id ndHtIqPtJoLR for ; Tue, 5 Nov 2013 07:16:46 -0800 (PST) Received: from co300216-co-outbound.net.avaya.com (co300216-co-outbound.net.avaya.com [198.152.13.100]) by ietfa.amsl.com (Postfix) with ESMTP id 46D7C11E80FB for ; Tue, 5 Nov 2013 07:16:42 -0800 (PST) X-IronPort-Anti-Spam-Filtered: true X-IronPort-Anti-Spam-Result: Ap4GAA0LeVLGmAcV/2dsb2JhbABZgmYhgQu/N4EsFm0HgicBAQMSKFEBFRUUQh8HAQQbGodfAZ0UhEidBI8og1iBDwOeboslgyaCKg X-IronPort-AV: E=Sophos;i="4.93,640,1378872000"; d="scan'208";a="35322984" Received: from unknown (HELO co300216-co-erhwest-exch.avaya.com) ([198.152.7.21]) by co300216-co-outbound.net.avaya.com with ESMTP; 05 Nov 2013 10:16:42 -0500 Received: from unknown (HELO AZ-FFEXHC03.global.avaya.com) ([135.64.58.13]) by co300216-co-erhwest-out.avaya.com with ESMTP; 05 Nov 2013 10:11:11 -0500 Received: from AZ-FFEXMB04.global.avaya.com ([fe80::6db7:b0af:8480:c126]) by AZ-FFEXHC03.global.avaya.com ([135.64.58.13]) with mapi id 14.03.0146.000; Tue, 5 Nov 2013 10:16:40 -0500 From: "Romascanu, Dan (Dan)" To: "lmap@ietf.org" Thread-Topic: meeting tomorrow Thread-Index: Ac7aOgYRzOa+pveqR5G2rS5l3F+ODw== Date: Tue, 5 Nov 2013 15:16:40 +0000 Message-ID: <9904FB1B0159DA42B0B887B7FA8119CA1292C8E1@AZ-FFEXMB04.global.avaya.com> Accept-Language: en-US Content-Language: en-US X-MS-Has-Attach: X-MS-TNEF-Correlator: x-originating-ip: [135.64.58.46] Content-Type: text/plain; charset="us-ascii" Content-Transfer-Encoding: quoted-printable MIME-Version: 1.0 Subject: [lmap] meeting tomorrow X-BeenThere: lmap@ietf.org X-Mailman-Version: 2.1.12 Precedence: list List-Id: Large Scale Measurement of Access network Performance List-Unsubscribe: , List-Archive: List-Post: List-Help: List-Subscribe: , X-List-Received-Date: Tue, 05 Nov 2013 15:16:55 -0000 Hi, LMAP is meeting tomorrow, Wednesday at 13:00 in Georgia A.=20 We are still looking for volunteers to take minutes and keep jabber connect= ivity with the remote participants. Without them we cannot start the meetin= g.=20 Also, all presenters please send your slides before the end of today, so th= at we can upload and make available for the remote participants.=20 Thanks and Regards, Jason and Dan From philip.eardley@bt.com Tue Nov 5 18:43:22 2013 Return-Path: X-Original-To: lmap@ietfa.amsl.com Delivered-To: lmap@ietfa.amsl.com Received: from localhost (localhost [127.0.0.1]) by ietfa.amsl.com (Postfix) with ESMTP id 79CBE21F9DA1 for ; Tue, 5 Nov 2013 18:43:22 -0800 (PST) X-Virus-Scanned: amavisd-new at amsl.com X-Spam-Flag: NO X-Spam-Score: -103.334 X-Spam-Level: X-Spam-Status: No, score=-103.334 tagged_above=-999 required=5 tests=[AWL=0.265, BAYES_00=-2.599, RCVD_IN_DNSWL_LOW=-1, USER_IN_WHITELIST=-100] Received: from mail.ietf.org ([12.22.58.30]) by localhost (ietfa.amsl.com [127.0.0.1]) (amavisd-new, port 10024) with ESMTP id DNY8wBn-AFEL for ; Tue, 5 Nov 2013 18:43:13 -0800 (PST) Received: from smtpe1.intersmtp.com (smtp63.intersmtp.com [62.239.224.236]) by ietfa.amsl.com (Postfix) with ESMTP id F351E21E815D for ; Tue, 5 Nov 2013 18:42:11 -0800 (PST) Received: from EVMHT67-UKRD.domain1.systemhost.net (10.36.3.104) by RDW083A007ED63.smtp-e3.hygiene.service (10.187.98.12) with Microsoft SMTP Server (TLS) id 8.3.298.1; Wed, 6 Nov 2013 02:42:10 +0000 Received: from EMV67-UKRD.domain1.systemhost.net ([169.254.2.147]) by EVMHT67-UKRD.domain1.systemhost.net ([10.36.3.104]) with mapi; Wed, 6 Nov 2013 02:42:10 +0000 From: To: , Date: Wed, 6 Nov 2013 02:42:10 +0000 Thread-Topic: LMAP regulator use case Thread-Index: Ac7VfioGb4cSexizQzSZrQa31mFanAFFb7bP Message-ID: References: <793D91975B99224DA9777562FF192808A27468CB@exmbx01> In-Reply-To: <793D91975B99224DA9777562FF192808A27468CB@exmbx01> Accept-Language: en-US, en-GB Content-Language: en-GB X-MS-Has-Attach: X-MS-TNEF-Correlator: acceptlanguage: en-US, en-GB Content-Type: text/plain; charset="iso-8859-1" Content-Transfer-Encoding: quoted-printable MIME-Version: 1.0 Subject: Re: [lmap] LMAP regulator use case X-BeenThere: lmap@ietf.org X-Mailman-Version: 2.1.12 Precedence: list List-Id: Large Scale Measurement of Access network Performance List-Unsubscribe: , List-Archive: List-Post: List-Help: List-Subscribe: , X-List-Received-Date: Wed, 06 Nov 2013 02:43:22 -0000 Frode thanks very much for the comments - and it is excellent to have detailed te= xt suggestions. 2.2 I think i'm basically OK with this replacing the current S2.2 (including th= e subsections) I suggest that it should mention comparability in the final para (as well a= s in the penultimate one) - personally I think this is the most important c= haracteristic from a regulator's point of view, so that it's fair to compar= e measurements they make across the different ISPs. the one thing in the current S2.2 (actually in S2.2.3) that I think is wort= h keeping is a mention of fixed and mobile. I think this could be handled b= y adding to the Intro (S1) something like: "Measurements will take place on= both fixed and mobile access" Happy to add a new chapter 4, but comments about some of your sub use cases 4.1 Promoting competition through transparency >>For competition to successfully discipline operators' behaviour in the in= terests of their customers, end users must be fully aware of the characteri= stics of the ISPs' access offers. "must be fully aware" over-states things. >>The system can be operated by a regulator or a measurement provider. add "on behalf of the regulator" >> However, the quality of an Internet access service is also determined b= y the connectivity to the rest of the Internet. Therefore test configuratio= ns which measure beyond the ISP's own network may also be relevant in some = cases. true. perhaps worth mentioning that the ISPs can influence this to some ext= ent - how well they're connected to the Internet, how they cache popular co= ntent etc. 4.2 Promoting end user empowerment i think this can be deleted. this is the end user use case, not the regulat= or use case. 4.3 Monitoring overall market development good for me 4.4 Detecting degradation of the Internet access 4.5 Detecting degradation of specific applications I don't like these sections about testing for whether the ISP is being net = neutral. I think it would be very hard to design tests. In part this is because net neutrality is more of a political concept than = technical one (in my view) i tihnk some of the stuff in S4.4 is really part of 4.3, as it's a market d= evelopment issue (the internet has got slower, and doesnt meet the policy o= bjective of Superfast broadband by Year 2015 - or whatever)- best wishes phil ________________________________________ From: lmap-bounces@ietf.org [lmap-bounces@ietf.org] On Behalf Of S=F8rensen= , Frode [frode.sorensen@npt.no] Sent: 30 October 2013 14:55 To: lmap@ietf.org Subject: [lmap] LMAP regulator use case Marc, Phil, Trevor, all, I have prepared a proposed description of a regulator use case for the LMAP= use case ID. I noticed the "advertisement" for further input from regulato= rs for the use case a couple of weeks ago. Finally I have managed to find s= ufficient time to contribute to the ID, supported by a couple of "regulator= friends" from the Net Neutrality Working Group of the BEREC organization, = representing European national regulators. The proposal consists of two par= ts; a modification of section 2.2 "Regulators", plus a new chapter after ch= apter 3 "Details of ISP use case", titled "Details of regulator use case". Thanks, Frode Proposed update: 2.2 Regulators ------------------- Regulators in jurisdictions around the world are responding to consumers' a= doption of Internet access services for traditional telecommunications and = media services by promoting competition among providers of electronic commu= nications, to ensure that users derive maximum benefit in terms of choice, = price, and quality. Some jurisdictions have responded to a need for greater information about I= nternet access service performance in the development of regulatory policie= s and approaches for broadband technologies by developing large-scale measu= rement programs. Programs such as the U.S. Federal Communications Commissio= n's Measuring Broadband America, European Commission's Quality of Broadband= Services in the EU reports and a growing list of other programs employ a d= iverse set of operational and technical approaches to gathering data to per= form analysis and reporting on diverse aspects of broadband performance. While each jurisdiction responds to distinct consumer, industry, and regula= tory concerns, much commonality exists in the need to produce datasets that= are able to compare multiple Internet access service providers, diverse te= chnical solutions, geographic and regional distributions, and marketed and = provisioned levels and combinations of broadband Internet access services. = In some jurisdictions, the role of measuring is provided by a measurement p= rovider. Measurement providers measure network performance from users towards multip= le content and application providers, included dedicated test measurement s= ervers, to show a performance of the actual Internet access service provide= d by different ISPs. Users need to know the performance that are achieving = from their own ISP. In addition, they need to know the performance of other= ISPs of same location as background information for selecting their ISP. M= easurement providers will provide measurement results with associated measu= rement methods and measurement metrics. >From a consumer perspective, the differentiation between fixed and mobile (= cellular) Internet access services is blurring as the applications used are= very similar. Hence, similar measurements will take place on both fixed an= d mobile Internet access services. Regulators role in the development and enforcement of broadband Internet ac= cess service policies also require that the measurement approaches meet a h= igh level of verifiability, accuracy and provider-independence to support v= alid and meaningful comparisons of Internet access service performance LMAP standards could answer regulators shared needs by providing scalable, = cost-effective, scientifically robust solutions to the measurement and coll= ection of broadband Internet access service performance information. Proposed new chapter: 4 Details of regulator use case --------------------------------------- 4.1 Promoting competition through transparency --------------------------------------------------------------- Competition plays a vital role in regulation of the electronic communicatio= ns markets. For competition to successfully discipline operators' behaviour= in the interests of their customers, end users must be fully aware of the = characteristics of the ISPs' access offers. In some jurisdictions regulator= s mandate transparent information made available about service offers. End users need effective transparency to be able to make informed choices t= hroughout the different stages of their relationship with ISPs, when select= ing Internet access service offers, and when considering switching service = offer within an ISP or to an alternative ISP. Quality information about ser= vice offers can be provided based on performance measurements of the servic= es provided in the market. A set of measurement parameters and associated measurement methods are used= over time, e.g. speed, delay, and jitter. Then the measurement raw data ar= e collected and go through statistical post-processing before the results c= an be published in an Internet access service quality index to facilitate e= nd users' choice of service provider and offer. In order to make information on offerings more meaningful and comparable to= end users, common frames of reference on Internet access service quality m= etrics are essential. An example would be that in relation to access speed,= the actual download and upload speeds are measured and published, not only= the maximum speed. A measurement system that monitor Internet access services and collect qual= ity information can typically consist of a number of measurement probes and= one or more test servers located at peering points. The system can be oper= ated by a regulator or a measurement provider. Number and distribution of = probes follows specific requirements depending on the scope and the desired= statistical reliability of the measurement campaign. In many test configurations, the measurement results will only reflect the = performance of ISP's own network. However, the quality of an Internet acces= s service is also determined by the connectivity to the rest of the Interne= t. Therefore test configurations which measure beyond the ISP's own network= may also be relevant in some cases. 4.2 Promoting end user empowerment -------------------------------------------------- In addition to the advantages that end users receive from the competition i= n the market, through availability of a plethora of ISPs and Internet acces= s service offers, facilitated by transparent quality information about thes= e offers at a general level, end users would further leverage on informatio= n about specific services they subscribe to, or which they are considering = to subscribe to. Performance of Internet access service offers vary based on where the servi= ce is provided (geographical and topological aspects), how it is provided (= e.g. access technology and ISP's interconnection to the Internet) and when = it is provided (variation through the day and over the week). Therefore the= re is a need to obtain more detailed quality information about specific ser= vice offers. Quality measurement tools can be made available for end users to monitor th= eir Internet access service, giving information about the speed and other q= uality metrics of the subscribed service. Speed meters and similar tools ca= n be provided by regulators or other third parties, but also directly by th= e ISP for the benefit of the subscriber. Such measurement tools assist end users in verifying the performance of the= Internet access service provided, comparing the measurement results achiev= ed with the information given in the contract. If the contracted service qu= ality level is not met, end users can use these results when complaining to= the ISP to achieve improvement and compensation. It may also help end users to assess whether they have chosen the package w= hich best suits their specific needs. With their performance results at han= d, they can benefit from quality evaluation guides, which offer comparisons= based on their specific usage profiles. This may enable them to check whet= her their current subscription is the most appropriate, and assist them to = switch package or provider if it is in their best interest. When provided by regulators and other third parties, such quality measureme= nt tools can also be used to compare performance between ISPs. It should, t= hough, be recognised that measurement results can be affected by factors ou= tside the ISP's control, related for instance with local network connection= (e.g. Wi-Fi), equipment and software being used. 4.3 Monitoring overall market development --------------------------------------------------------- Governments sometimes set strategic goals for high-speed Internet access pe= netration as an important component of the economic, cultural and social de= velopment of the society. To evaluate the effect of the stimulated growth o= ver time, broadband Internet access take-up and penetration of high-speed a= ccess can be monitored through measurement campaigns. An example of such an initiative is the "Digital Agenda for Europe" which w= as adopted in 2010, to achieve universal broadband access. The goal is to a= chieve by 2020, access for all Europeans to Internet access speeds of 30 Mb= ps or above, and 50% or more of European households subscribing to Internet= connections above 100 Mbps. To monitor actual broadband Internet access performance in a specific count= ry or a region, extensive measurement campaigns are needed. A panel can be = built based on operators and packages in the market, spread over urban, sub= urban and rural areas. Probes can then be distributed to the participants o= f the campaign. Periodic tests running on the probes can for example measure actual speed a= t peak and off-peak hours, but also other detailed quality metrics like del= ay and jitter. Collected data goes afterwards through statistical analysis,= deriving estimates for the whole population which can then be presented an= d published regularly. Using a harmonized or standardised measurement methodology, or even a commo= n quality measurement platform, measurement results could also be used for = benchmarking of providers and/or countries. 4.4 Detecting degradation of the Internet access -------------------------------------------------------------- Regulation related to net neutrality and the open Internet has been introdu= ced in some jurisdictions' Internet policy, and monitoring methods for dete= ction of potential net neutrality breaches is needed to follow up such regu= lations with concrete actions. Net neutrality is typically related to equal treatment of traffic transmitt= ed over the Internet access service, while other traffic transmitted in par= allel over the end user's broadband connection e.g. to provide IPTV in IP n= etworks separated from the Internet (also referred to as "closed IP network= s") can be exempted from net neutrality regulation. Services provided in pa= rallel to the Internet access services are referred to as specialised servi= ces (ref. BEREC and FCC). In order for such regulatory service architecture to protect net neutrality= and the open Internet, it is essential that the specialised services are n= ot provided at the expense of the Internet access service. Therefore regula= tors may need to measure performance of the Internet access service over ti= me, and thereby detect whether this service is becoming degraded. When regulators monitor this development to ensure net neutrality and the o= pen Internet, they can check e.g. whether Internet access service performan= ce and level of quality reflect advances in technology and that this is not= impaired by specialised services. Comparison between ISPs or between diffe= rent countries may also be relevant for this kind of evaluation. Based on such Internet access service quality monitoring, regulators will t= ypically report results and findings regularly, e.g. on an annual basis. In= some jurisdictions regulators may also eventually impose minimum quality o= f service requirements on ISPs in order to prevent degradation of Internet = access services. 4.5 Detecting degradation of specific applications --------------------------------------------------------------- In the regulatory context related to net neutrality and the open Internet, = in addition to the monitoring to detect potential degradation of the access= service as a whole described above, it is essential to perform monitoring = to detect potential degradation of individual applications using the Intern= et access. Since net neutrality relates to equal treatment of traffic transmitted over= the Internet access service, the treatment of different applications' traf= fic is in the core of net neutrality. When traffic originating from differe= nt applications are treated independent of which application each IP packet= belongs to, this is referred to as application agnosticism. Examples of departure from application agnosticism are blocking or throttli= ng of traffic from specific applications, but also preferential treatment o= f specific applications. If traffic is forwarded at different priority leve= ls, traffic having higher priority level implies that other traffic has low= er priority. Detection of such traffic management practices does not necess= arily imply breaches to net neutrality, but these observations can be used = as an input to the regulatory evaluation of the practice. When regulators monitor traffic management practices to ensure net neutrali= ty and the open Internet, they can check compliance with criteria defining = what is considered reasonable traffic management. Measurement systems desig= ned to monitor this kind of effects need to send application-specific traff= ic and then measure in detail the behaviour of the different packets receiv= e when transferred over the Internet access service. Today there exist measurement tools that can detect differentiated treatmen= t of individual applications. Another method that can be used is to measure= performance at application layer, e.g. assisted by content and application= providers, but also passive measurements are foreseen as a promising appro= ach for monitoring of application-specific treatment. _______________________________________________ lmap mailing list lmap@ietf.org https://www.ietf.org/mailman/listinfo/lmap From steve@idrathernotsay.com Tue Nov 5 20:11:37 2013 Return-Path: X-Original-To: lmap@ietfa.amsl.com Delivered-To: lmap@ietfa.amsl.com Received: from localhost (localhost [127.0.0.1]) by ietfa.amsl.com (Postfix) with ESMTP id 47A8221E80DC for ; Tue, 5 Nov 2013 20:11:37 -0800 (PST) X-Virus-Scanned: amavisd-new at amsl.com X-Spam-Flag: NO X-Spam-Score: -0.185 X-Spam-Level: X-Spam-Status: No, score=-0.185 tagged_above=-999 required=5 tests=[BAYES_40=-0.185] Received: from mail.ietf.org ([12.22.58.30]) by localhost (ietfa.amsl.com [127.0.0.1]) (amavisd-new, port 10024) with ESMTP id ynLTp9C5dUGX for ; Tue, 5 Nov 2013 20:11:32 -0800 (PST) Received: from colo3.idrathernotsay.com (colo3.idrathernotsay.com [74.207.251.199]) by ietfa.amsl.com (Postfix) with ESMTP id B7A8A21E80CA for ; Tue, 5 Nov 2013 20:11:32 -0800 (PST) Received: from localhost (localhost [127.0.0.1]) by colo3.idrathernotsay.com (Postfix) with ESMTP id BC857FC4E for ; Wed, 6 Nov 2013 04:11:32 +0000 (UTC) X-Virus-Scanned: amavisd-new at idrathernotsay.com Received: from colo3.idrathernotsay.com ([127.0.0.1]) by localhost (colo3.idrathernotsay.com [127.0.0.1]) (amavisd-new, port 10024) with ESMTP id iSZXxFL92IFm for ; Wed, 6 Nov 2013 04:11:29 +0000 (UTC) Received: from newbie.idrathernotsay.com (pool-173-79-225-215.washdc.fios.verizon.net [173.79.225.215]) by colo3.idrathernotsay.com (Postfix) with ESMTPSA id EF44BFC3B for ; Wed, 6 Nov 2013 04:11:28 +0000 (UTC) Received: from newbie.idrathernotsay.com (localhost [127.0.0.1]) by newbie.idrathernotsay.com (Postfix) with ESMTP id 079B34FBCF4; Tue, 5 Nov 2013 23:11:28 -0500 (EST) X-Virus-Scanned: amavisd-new at idrathernotsay.com Received: from newbie.idrathernotsay.com ([127.0.0.1]) by newbie.idrathernotsay.com (newbie.idrathernotsay.com [127.0.0.1]) (amavisd-new, port 10024) with ESMTP id CCpaNuyIxkML; Tue, 5 Nov 2013 23:11:26 -0500 (EST) Received: by newbie.idrathernotsay.com (Postfix, from userid 1000) id 804AB4FBD05; Tue, 5 Nov 2013 23:11:26 -0500 (EST) Date: Tue, 5 Nov 2013 23:11:26 -0500 From: Steve Miller To: lmap@ietf.org Message-ID: <20131106041126.GA57287@idrathernotsay.com> MIME-Version: 1.0 Content-Type: text/plain; charset=us-ascii Content-Disposition: inline User-Agent: Mutt/1.5.21 (2010-09-15) Subject: [lmap] Comments on the current round of I-Ds, before the meeting tomorrow X-BeenThere: lmap@ietf.org X-Mailman-Version: 2.1.12 Precedence: list List-Id: Large Scale Measurement of Access network Performance List-Unsubscribe: , List-Archive: List-Post: List-Help: List-Subscribe: , X-List-Received-Date: Wed, 06 Nov 2013 04:11:37 -0000 Hi, I'm Steve Miller, and (despite the email address (-: ) I work for Verisign. I've been lurking on the lmap list for quite a while, and while I have some background in working on systems such as the ones being discussed here, I haven't had a chance until the flight out to Vancouver to really dig into the current crop of I-Ds. I figured I'd send my comments out here, as while I have a fair number of them, it's not clear to me that any of them are necessarily worth my tying up a microphone during the WG meeting. I'll start by saying that I'm excited to see a group tackling this sort of large-scale performance-measurement problem. I think this sort of thing has huge potential to be useful to ISPs, to content providers, and (directly or indirectly) to the end users. And I'll say that the answer "go read the mail archive, we've covered all this" is also acceptable. (-: I'm in that unenviable position where I feel like I'm behind on all that's transpired, so while I feel like I've done a lot of my homework I doubt I've done it all. I hope some of what I say makes sense and that the places where I screw up, I don't screw up too much... Before I work through the drafts, I had some general notes about time. Time handling seems to be one of those things that shouldn't be all that hard, but given how many implementors I see get it wrong, I'm thinking it'd be good if we can stack the odds a bit in our favor. I'd think very seriously about changing all the timestamp stuff in here to be seconds and microseconds since midnight 1970 UTC (the usual Unix-like timestamp, though of course going to nanoseconds wouldn't be tragic). I'd also think about enforcing the idea that all the MAs and the collectors (and probably the controllers) all express and interpret all times as UTC, as otherwise -- at least in a system where more than one organization may be using the same set of MAs, or where the set of MAs is made up of systems controlled by more than one organization -- we'll end up with half the stuff in UTC and a quarter in EST and some in every other timezone out there. Finally, it might be worth calling out that all these systems need to have their clocks synced using NTP or GPS signals or something roughly equivalent, or the users of such a system will tear their hair out trying to cross-correlate results from different MAs. It is possible that this sort of scenario is responsible for some of my own hair loss. Now, working through the drafts in order from the PDF in the agenda... LMAP Protocol (draft-bagnulo-lmap-http-01) ------------------------------------------ I see several places in the examples, at least, where the MA is POSTing its IP address: e.g., one where it's posting that as part of a dump of interface information, and one where it's posting it as part of the results for the sample UDP latency check. For a MA that's sitting at the boundary of a network (e.g., the router in some end-user's house), that's not too bad to do, but for a MA that's further inside such a network, it might be difficult for it to know what "real" (non-RFC-1918) address it's using. I could definitely see how having it publish the real address that it used would be useful to the collector but if we need that we'll have to figure out how to have the MA discover the real address it'd be using. I'm not sure how useful it is to just have the RFC1918 address, though. It occurred to me, too, thinking more about the case where the MA is inside a network rather than at its boundary, saying stuff about the interface speed should be taken with a grain of salt. After all, if the MA says it's connected at GigE, that's great, but if it's GigE up to the DSL line, we'll see some possibly-surprising test results. When it comes to handling communications failures (6.3.1), there's all the usual stuff to worry about in terms of what to do about a more prolonged communications failure. I could see arguments why the Controller would want all the resultsbut "all" can be a dangerous word. I think we could also make an argument that the results of measurement X are not as interesting once the results for measurement X+1 of the same type are available. Should we issue any guidance? Or just assume that this is the sort of detail that the implementors will get right? There's a fair amount of discussion about exponential backoff but not anything to say when to stop. (-: FWIW, with respect to not doing tests when there's cross-traffic, or at least when it comes to indicating that there is cross traffic, in some similar things I've worked on I've relied on tracking the results on a per-MA basis as a hint whether the suitability of a given MA has been compromised. That is, if the performance for a given MA for a related set of checks has been at a certain level and then changes to a different level (more loss, higher latency, etc.), that's a strong hint as to the performance of the network in or around the MA. For that matter, there are many things that can happen between the MA and what it's testing, that could make the results from that MA look worse, so to some extent it'd be easy to look at the "no cross traffic, all awesome!" indicator and end up with a false sense of security. A lot of that comes down to the age-old problem in this problem space, when device X says there's a problem with Y, if Y is actually broken or if X is broken. All that being said, given that the MA may not be executing measurements solely on behalf of a single requestor I'm not sure that I see how to cross-correlate the results from a single MA. Information Model (draft-burbridge-lmap-information-model-01) ------------------------------------------------------------- About the security model (not really specific to this I-D, admittedly): the idea of using the group ID as a security measure, to make it harder to associate the results of a measurement with an individual MA, seems optimistic. Not that I have a better idea, mind you, but I did feel that I should say that. About periodic timing: why specify the interval in milliseconds and not, say, microseconds, or even just fractional seconds (e.g., 0.2)? Sure, I think the ability to launch tests with microsecond precision is a bit questionable, but I don't see how specifying greater precision really hurts us so long as people understand that the MA's ability to hit that precisely -- even on a millisecond basis -- is going to be somewhat limited. I'm not sure that I saw anything call out what a MA should do if it can't execute a measurement at the specified time (e.g., it's too busy with other work to do so). Another classic failure of simplistic implementations of systems like the MA is to assume that they can complete all work in the time allowed, and when inevitably that doesn't happen (e.g., every test takes much longer than usual because they all time out because the device has lost connectivity) they just get further and further behind. I think in that sort of scenario the MA should simply not execute any measurements that it can't do on time, alerting to its manager that it has an issue and if possible reporting to the relevant collector for each measurement it can't do that it couldn't do that measurement. Metrics Registry (draft-mornulo-ippm-registry-00) (and draft-mornulo-ippm-registry-columns-01) ------------------------------------------------- It might be good to have the idea captured somewhere, somehow that we could have measurements that return a histogram of their results or some other indication as to their distribution (e.g., a histogram of latencies in the "go do this 100 times and then tell me the results" case). It's important to express the concept of null values as well (which might be covered, I wasn't sure). I've seen lots of people try to use sentinel values to represent things like the latency in the case of 100% loss, and that tends to end up destroying information ("it's not zero, it's undefined!") and ultimately often ends up getting averaged inadvertantly with real data ("hey, this had 99% loss, but the latency is freakishly low"). It seemed like this was calling out the idea of expressing the registry as columns in what seems logically like a database or spreadsheet, but I like the idea of expressing it in JSON or YAML or something more like that. I may be reading this too literally here. LMAP Framework (draft-ietf-lmap-framework-01) --------------------------------------------- In general, the model here always has the MA connecting to the collector to submit the results of a measurement. I've run into situations before where it's been considered preferable to have the collector connect outbound to the MAs to get the relevant results. Do we have a way to address that? (Admittedly that's more difficult given that many MAs are likely to be behind NATs; we'd have to address that as well, even if we just pointed to suggestions as to how the MA could keep a line of communication open. I know that's more or less a solved problem even without resorting to uPNP or whatever, because I know that some of the VOIP vendors do it, even though the solutions are all kinda heedious.) This may be a little controversial to say, but for a system of significant scale it seems unlikely that using a SQL database for the Results Database will work as well as one might hope (ask me how I know). It might be worth calling that out, though that's a minor tweak. In 4.2, there's discussion about having just one controller at a time, but I thought the HTTP draft said or implied that a MA might have more than one controller telling it what to do. p. 14, at the top, discussion about what to do if the controller fails: I'm not sure that's all that important to consider here. It seems like the MA can just do another DNS lookup to locate the controller (and the owner of the controller can change the IP address returned for the controller if needed), or the owner of the controller can just stand up another host at the same IP address -- in short, that seems like a solved problem and in this case I'm not sure it's worth calling out the solutions. LMAP Use Cases (draft-ietf-lmap-use-cases-00) --------------------------------------------- There's a specific use case in which I have quite a lot of interest, and while the text that's here already certainly doesn't preclude that use case, I don't know if it'd be worth laying out the use case I envision, or if it's too special-case. I think there's an unspoken assumption here that a measurement is from point A to point B, when for certain types of measurements -- DNS being the first one that comes to my mind! (-: -- are from point A to one of potentially many points B, because B is anycasted. That's also not too different from the case where A finds B via a DNS lookup, but B has more than one IP address. Most, if not all, of [a-m].root-servers.net are anycasted, as are several of the com/net service addresses, and certainly Verisign's peers and competitors do the same sorts of things, they're standard operating procedure in the DNS world. Something that I think all operators have to deal with is that with heavily anycasted infrastructure, it's easy to end up in a situation where some users have poor connectivity to some service instance, but there's no good way for the operator of the infrastructure to *know* that some users are having issues. For example, if someone connected to an ISP in Cairo is having issues hitting the j.root-servers.net instance in Cairo, I don't have a good way to know that unless I can also test from within that user's ISP. So having the ability to perform a small number of tests from a very broad range of MAs with wide IP-topologic scope would be massively useful. Going along with that, we'd need to have some way as part of the measurement being performed to have the MA identify the instance it's reached -- including the result of a "chaos null hostname.bind" query, a traceroute, or somesuch, would be needed. Anyway, we might be weird, but since I think we have a lot of company, I thought it was worth calling out this use case... There's also a lot of emphasis in this and the other docs about backing off when the link is in use, but I think there's a lot to be learned from performing relatively low-bandwidth tests all the time, even when the link is otherwise in use, small enough that no real backoff approach is needed. You can learn a lot from pinging something every 10 seconds or so, without burning a lot of bandwidth. I agree that at least in certain scenarios we'd have to worry about counting against someone's data allowance. There's some stuff in here about aggregation, and in particular about using aggregation in part to help preserve user privacy. I'm not sure that I'm personally all that keen on aggregation; it seems like if you do short-term aggregation, inevitably you say "hey this thing is busted!" and then you want the details, which of course by then are gone. In the longer term, I also think it's easy to get into situations where there's something odd with the data, or you need to lookat it in a different way, and with aggregated data it can be difficult to track down the source of (for example) data-collection problems that happened months ago. I understand the desire to aggregate, from both a privacy standpoint and because it'd be nice not to buy disks by the truckload, but I wanted to call this out as well. -Steve From Klaus.Nieminen@ficora.fi Tue Nov 5 21:19:25 2013 Return-Path: X-Original-To: lmap@ietfa.amsl.com Delivered-To: lmap@ietfa.amsl.com Received: from localhost (localhost [127.0.0.1]) by ietfa.amsl.com (Postfix) with ESMTP id A386D21E80E1 for ; Tue, 5 Nov 2013 21:19:25 -0800 (PST) X-Virus-Scanned: amavisd-new at amsl.com X-Spam-Flag: NO X-Spam-Score: -2.599 X-Spam-Level: X-Spam-Status: No, score=-2.599 tagged_above=-999 required=5 tests=[BAYES_00=-2.599] Received: from mail.ietf.org ([12.22.58.30]) by localhost (ietfa.amsl.com [127.0.0.1]) (amavisd-new, port 10024) with ESMTP id ME3O9MNBqdhG for ; Tue, 5 Nov 2013 21:19:21 -0800 (PST) Received: from out2.ficora.fi (out2.ficora.fi [87.239.124.62]) by ietfa.amsl.com (Postfix) with ESMTP id 9279D21E80B1 for ; Tue, 5 Nov 2013 21:19:19 -0800 (PST) X-IronPort-AV: E=Sophos;i="4.93,644,1378846800"; d="scan'208";a="8287517" Received: from LOOTA.laru.local ([fe80::3cb5:71aa:ca40:2a8d]) by loota.laru.local ([fe80::3cb5:71aa:ca40:2a8d%10]) with mapi id 14.03.0158.001; Wed, 6 Nov 2013 07:19:18 +0200 From: Nieminen Klaus To: "philip.eardley@bt.com" , "frode.sorensen@npt.no" , "lmap@ietf.org" Thread-Topic: [lmap] LMAP regulator use case Thread-Index: AQHO2qZaoBFaE8+BzkaRHmy3dVQTU5oXnIIw Date: Wed, 6 Nov 2013 05:19:16 +0000 Message-ID: <6AF4522BD5AB86429CFD3948A9AB2F2F3A22854A@loota.laru.local> References: <793D91975B99224DA9777562FF192808A27468CB@exmbx01> In-Reply-To: Accept-Language: en-US, fi-FI Content-Language: en-US X-MS-Has-Attach: X-MS-TNEF-Correlator: x-originating-ip: [87.239.126.79] Content-Type: text/plain; charset="Windows-1252" Content-Transfer-Encoding: quoted-printable MIME-Version: 1.0 Subject: Re: [lmap] LMAP regulator use case X-BeenThere: lmap@ietf.org X-Mailman-Version: 2.1.12 Precedence: list List-Id: Large Scale Measurement of Access network Performance List-Unsubscribe: , List-Archive: List-Post: List-Help: List-Subscribe: , X-List-Received-Date: Wed, 06 Nov 2013 05:19:25 -0000 Hi Philip, Frode and all, As one of the regulator friends that Frode mentioned, I would like to tell= how we see these issues in Finland as I'm working for the Finnish telco re= gulator. 4.2 Promoting end user empowerment I would like to see this chapter to be included. As said publishing transpa= rent and comparable information is important but so it's also end user's ab= ility to verify the performance of the Internet access service provided, co= mparing the measurement results achieved with the information given in the = contract. If the contracted service quality level is not met, end users can= use these results when complaining to the ISP to achieve improvement and c= ompensation. If you want to include it to end user requirements it is ok for me to refer= ence it from the regulator part. However I do not want it to be totally del= eted as it is also our requirement not only end user one. 4.4 Detecting degradation of the Internet access 4.5 Detecting degradation of specific applications Yes I know that this is a hard problem how to measure NN. However it is a c= lear regulatory requirement at least in the EU. See Directive 2009/136/EC h= ttp://eur-lex.europa.eu/LexUriServ/LexUriServ.do?uri=3DOJ:L:2009:337:0011:0= 1:EN:HTML=20 According to Art 22(3), =93in order to prevent the degradation of service a= nd the hindering or slowing down of traffic over networks, Member States sh= all ensure that national regulatory authorities are able to set minimum qua= lity of service requirements on an undertaking or undertakings providing pu= blic communications networks.=94 The reach of this regulatory tool has been= extensively examined by BEREC, which sees it as the third and last step of= the regulatory approach to net neutrality breaches So we do need tools for this purpose as Frode has described. best regards, - Klaus - ___________________________________________ KLAUS NIEMINEN Communications Network Specialist Finnish Communications Regulatory Authority P.O. Box 313 FIN-00181 HELSINKI, FINLAND tel.: +358 29 5390 528 fax: +358 9 6966 873 e-mail: klaus.nieminen@ficora.fi=20 www: http://www.ficora.fi =20 -----Original Message----- From: philip.eardley@bt.com [mailto:philip.eardley@bt.com]=20 Sent: 6. marraskuuta 2013 4:42 To: frode.sorensen@npt.no; lmap@ietf.org Subject: Re: [lmap] LMAP regulator use case Frode thanks very much for the comments - and it is excellent to have detailed te= xt suggestions. 2.2 I think i'm basically OK with this replacing the current S2.2 (including th= e subsections) I suggest that it should mention comparability in the final = para (as well as in the penultimate one) - personally I think this is the m= ost important characteristic from a regulator's point of view, so that it's= fair to compare measurements they make across the different ISPs. the one thing in the current S2.2 (actually in S2.2.3) that I think is wort= h keeping is a mention of fixed and mobile. I think this could be handled b= y adding to the Intro (S1) something like: "Measurements will take place on= both fixed and mobile access" Happy to add a new chapter 4, but comments about some of your sub use cases 4.1 Promoting competition through transparency >>For competition to successfully discipline operators' behaviour in the in= terests of their customers, end users must be fully aware of the characteri= stics of the ISPs' access offers. "must be fully aware" over-states things. >>The system can be operated by a regulator or a measurement provider. add "on behalf of the regulator" >> However, the quality of an Internet access service is also determined b= y the connectivity to the rest of the Internet. Therefore test configuratio= ns which measure beyond the ISP's own network may also be relevant in some = cases. true. perhaps worth mentioning that the ISPs can influence this to some ext= ent - how well they're connected to the Internet, how they cache popular co= ntent etc. 4.2 Promoting end user empowerment i think this can be deleted. this is the end user use case, not the regulat= or use case. 4.3 Monitoring overall market development good for me 4.4 Detecting degradation of the Internet access 4.5 Detecting degradation of specific applications I don't like these sections about testing for whether the ISP is being net = neutral. I think it would be very hard to design tests. In part this is because net neutrality is more of a political concept than = technical one (in my view) i tihnk some of the stuff in S4.4 is really part of 4.3, as it's a market d= evelopment issue (the internet has got slower, and doesnt meet the policy o= bjective of Superfast broadband by Year 2015 - or whatever)- best wishes phil ________________________________________ From: lmap-bounces@ietf.org [lmap-bounces@ietf.org] On Behalf Of S=F8rensen= , Frode [frode.sorensen@npt.no] Sent: 30 October 2013 14:55 To: lmap@ietf.org Subject: [lmap] LMAP regulator use case Marc, Phil, Trevor, all, I have prepared a proposed description of a regulator use case for the LMAP= use case ID. I noticed the "advertisement" for further input from regulato= rs for the use case a couple of weeks ago. Finally I have managed to find s= ufficient time to contribute to the ID, supported by a couple of "regulator= friends" from the Net Neutrality Working Group of the BEREC organization, = representing European national regulators. The proposal consists of two par= ts; a modification of section 2.2 "Regulators", plus a new chapter after ch= apter 3 "Details of ISP use case", titled "Details of regulator use case". Thanks, Frode Proposed update: 2.2 Regulators ------------------- Regulators in jurisdictions around the world are responding to consumers' a= doption of Internet access services for traditional telecommunications and = media services by promoting competition among providers of electronic commu= nications, to ensure that users derive maximum benefit in terms of choice, = price, and quality. Some jurisdictions have responded to a need for greater information about I= nternet access service performance in the development of regulatory policie= s and approaches for broadband technologies by developing large-scale measu= rement programs. Programs such as the U.S. Federal Communications Commissio= n's Measuring Broadband America, European Commission's Quality of Broadband= Services in the EU reports and a growing list of other programs employ a d= iverse set of operational and technical approaches to gathering data to per= form analysis and reporting on diverse aspects of broadband performance. While each jurisdiction responds to distinct consumer, industry, and regula= tory concerns, much commonality exists in the need to produce datasets that= are able to compare multiple Internet access service providers, diverse te= chnical solutions, geographic and regional distributions, and marketed and = provisioned levels and combinations of broadband Internet access services. = In some jurisdictions, the role of measuring is provided by a measurement p= rovider. Measurement providers measure network performance from users towards multip= le content and application providers, included dedicated test measurement s= ervers, to show a performance of the actual Internet access service provide= d by different ISPs. Users need to know the performance that are achieving = from their own ISP. In addition, they need to know the performance of other= ISPs of same location as background information for selecting their ISP. M= easurement providers will provide measurement results with associated measu= rement methods and measurement metrics. From jamesmilleresquire@gmail.com Tue Nov 5 22:48:11 2013 Return-Path: X-Original-To: lmap@ietfa.amsl.com Delivered-To: lmap@ietfa.amsl.com Received: from localhost (localhost [127.0.0.1]) by ietfa.amsl.com (Postfix) with ESMTP id 0F8A611E8128 for ; Tue, 5 Nov 2013 22:48:11 -0800 (PST) X-Virus-Scanned: amavisd-new at amsl.com X-Spam-Flag: NO X-Spam-Score: -2.599 X-Spam-Level: X-Spam-Status: No, score=-2.599 tagged_above=-999 required=5 tests=[BAYES_00=-2.599, HTML_MESSAGE=0.001, NO_RELAYS=-0.001] Received: from mail.ietf.org ([12.22.58.30]) by localhost (ietfa.amsl.com [127.0.0.1]) (amavisd-new, port 10024) with ESMTP id KZ+d09JJ4VNY for ; Tue, 5 Nov 2013 22:48:08 -0800 (PST) Received: from mail-we0-x232.google.com (mail-we0-x232.google.com [IPv6:2a00:1450:400c:c03::232]) by ietfa.amsl.com (Postfix) with ESMTP id DD6B111E80F8 for ; Tue, 5 Nov 2013 22:48:07 -0800 (PST) Received: by mail-we0-f178.google.com with SMTP id q59so4413520wes.23 for ; Tue, 05 Nov 2013 22:48:06 -0800 (PST) DKIM-Signature: v=1; a=rsa-sha256; c=relaxed/relaxed; d=gmail.com; s=20120113; h=mime-version:in-reply-to:references:from:date:message-id:subject:to :cc:content-type; bh=ASYh5zQzZGtfttrGogY5Gnz2nTZe1dcxD+/yF/fq/vY=; b=XQ2/FHHNogGpPohn+2e0HtRSKYbmAaIHGm7tLH/uU4gMWDLRHCdi1Smbt/ucxSU/83 5eCUjfNMypxhY4mW4OrQgEXF3Ap2rgvfEWrBpPg6hp8Mnzy2JSNsUEUDXa3G48ZiDYuZ ex03diBWyUnZGnZQQHu0mt/Cu+hQJ/bRuK2vy0HqpKZbHTm4LgRAyuUNMUNEmQCTNzFb VQd0JNfVlILB1qNrBVOiSvJcNV9LCvaeIwED7L+O9PJEORun1mHBmjJ5UBCacURuBOOt ugUvAeBGvDC3ODH2ovQz5RA+W3ZlIdURnPDxnQNJhGon3NFoA+5E46YoIfQy1wC8w3W4 +HuA== X-Received: by 10.180.73.40 with SMTP id i8mr19809730wiv.37.1383720486443; Tue, 05 Nov 2013 22:48:06 -0800 (PST) MIME-Version: 1.0 Received: by 10.180.211.7 with HTTP; Tue, 5 Nov 2013 22:47:26 -0800 (PST) In-Reply-To: <6AF4522BD5AB86429CFD3948A9AB2F2F3A22854A@loota.laru.local> References: <793D91975B99224DA9777562FF192808A27468CB@exmbx01> <6AF4522BD5AB86429CFD3948A9AB2F2F3A22854A@loota.laru.local> From: James Miller Date: Wed, 6 Nov 2013 01:47:26 -0500 Message-ID: To: Nieminen Klaus Content-Type: multipart/alternative; boundary=f46d0438907d10212b04ea7c889f Cc: Henning Schulzrinne , "philip.eardley@bt.com" , "lmap@ietf.org" , Marc Linsner , "frode.sorensen@npt.no" Subject: Re: [lmap] LMAP regulator use case X-BeenThere: lmap@ietf.org X-Mailman-Version: 2.1.12 Precedence: list List-Id: Large Scale Measurement of Access network Performance List-Unsubscribe: , List-Archive: List-Post: List-Help: List-Subscribe: , X-List-Received-Date: Wed, 06 Nov 2013 06:48:11 -0000 --f46d0438907d10212b04ea7c889f Content-Type: text/plain; charset=windows-1252 Content-Transfer-Encoding: quoted-printable I would propose adding in section 2 or 4 some text describing a regulators need to ensure that policies intended to protect the privacy interests of a potential tester can be enforced using features available in the system. Recognizing that http://www.ietf.org/id/draft-folks-lmap-framework-00.txtand in the lists there's a view that a single entity will be overseeing a large scale measurement, there is a clear need for a test infrastructure to include tools to enforce the privacy policies of the managing entity and coordinate collaborative activity. A desire to protect the privacy of testers contributing to our mobile measurement effort lead our work on the project, and our decision to adopt changes to our fixed technical infrastructure and data model to support a totally anonymous collection at the MA and communications between controllers, and through the collectors and the test results collected. I understand there has been discussion on this point in the past and we had included this in the Section 7 Security Considerations of our original I-D. In any event I think the regulator use case benefits from calling the point out specifically. http://tools.ietf.org/id/draft-schulzrinne-lmap-requirements-00.txt Proposed addition in Section 2.2: :: Regulators role in the development and enforcement of broadband Internet access service policies also require that the measurement approaches meet a high level of verifiability, accuracy and provider-independence to support valid and meaningful comparisons of Internet access service performance. Regulators also require that policies intended to protect the privacy interests of potential testers can be enforced using features available in the system. Naturally any use case is going to have some variation among the specific practices of the parties. On Klaus and Frode's comments (and excellent draft contributions thanks!) I would chime in to add that "Transparency" is a fundamental feature of the rules intended to protect the Open Internet (or network neutrality in other parlance). Our rule is intended to enable "consumers to make informed choices regarding use of such services" but also "for content, application, service, and device providers to develop, market, and maintain Internet offerings." 47 CFR 8.3 - Transparency. Whether protecting consumers is moved I think you find the thought and regulatory use case solidly grounded in the US approach as well. I would also suggest adding some text to emphasize that Transparency is also important to enabling development in the ecosystem. Proposed addition in Section 4.1: :: End users need effective transparency to be able to make informed choices throughout the different stages of their relationship with ISPs, when selecting Internet access service offers, and when considering switching service offer within an ISP or to an alternative ISP. Transparency in Internet performance information is also important to content, application, service, and device providers' to ensure they are able to develop, market, and maintain Internet offerings. Quality information about service offering can be provided based on performance measurements of the services provided in the market. On Klaus' comments on what in US jurisdictions might be understand as "blocking" or "discriminatory" conduct, I'd add that US law includes similar provisions. 47 C.F.R.=A7=A7 8.5, 8.7. Recognizing that these topic= s including "Quality of Service" are evolving and differ in many respects across jurisdictions, a high level description focused on the value of performance information to regulators in this regard may be helpful in the section. I would also breach another topic for future discussion on assessing quality on the test network itself. To maintain comparability and quality of the collected data, the architecture itself must be monitored with auditing of whether the infrastructure is functioning correctly, e.g. server health and connectivity. MA side is much easier to keep a handle on with special purpose probes (such as with our fixed "whitebox" approach) but server side checks on and between controllers is also a very important issue to keep an eye on. I don't have a specific proposal how to address this in an I-D right now but I'd note it as something any large scale measurement manager will need to address. -- JamesMillerEsquire@Gmail.com "Wovon man nicht sprechen kann, dar=FCber mu=DF man schweigen." (What we cannot speak about we must pass over in silence. ) Ludwig Josef Johann Wittgenstein On Wed, Nov 6, 2013 at 12:19 AM, Nieminen Klaus w= rote: > Hi Philip, Frode and all, > > As one of the regulator friends that Frode mentioned, I would like to > tell how we see these issues in Finland as I'm working for the Finnish > telco regulator. > > 4.2 Promoting end user empowerment > > I would like to see this chapter to be included. As said publishing > transparent and comparable information is important but so it's also end > user's ability to verify the performance of the Internet access service > provided, comparing the measurement results achieved with the information > given in the contract. If the contracted service quality level is not met= , > end users can use these results when complaining to the ISP to achieve > improvement and compensation. > > If you want to include it to end user requirements it is ok for me to > reference it from the regulator part. However I do not want it to be > totally deleted as it is also our requirement not only end user one. > > 4.4 Detecting degradation of the Internet access > 4.5 Detecting degradation of specific applications > > Yes I know that this is a hard problem how to measure NN. However it is a > clear regulatory requirement at least in the EU. See Directive 2009/136/E= C > http://eur-lex.europa.eu/LexUriServ/LexUriServ.do?uri=3DOJ:L:2009:337:001= 1:01:EN:HTML > > According to Art 22(3), =93in order to prevent the degradation of service > and the hindering or slowing down of traffic over networks, Member States > shall ensure that national regulatory authorities are able to set minimum > quality of service requirements on an undertaking or undertakings providi= ng > public communications networks.=94 The reach of this regulatory tool has = been > extensively examined by BEREC, which sees it as the third and last step o= f > the regulatory approach to net neutrality breaches > > So we do need tools for this purpose as Frode has described. > > best regards, > > - Klaus - > ___________________________________________ > KLAUS NIEMINEN > Communications Network Specialist > Finnish Communications Regulatory Authority > P.O. Box 313 > FIN-00181 HELSINKI, FINLAND > tel.: +358 29 5390 528 > fax: +358 9 6966 873 > e-mail: klaus.nieminen@ficora.fi > www: http://www.ficora.fi > > -----Original Message----- > From: philip.eardley@bt.com [mailto:philip.eardley@bt.com] > Sent: 6. marraskuuta 2013 4:42 > To: frode.sorensen@npt.no; lmap@ietf.org > Subject: Re: [lmap] LMAP regulator use case > > Frode > thanks very much for the comments - and it is excellent to have detailed > text suggestions. > > 2.2 > I think i'm basically OK with this replacing the current S2.2 (including > the subsections) I suggest that it should mention comparability in the > final para (as well as in the penultimate one) - personally I think this = is > the most important characteristic from a regulator's point of view, so th= at > it's fair to compare measurements they make across the different ISPs. > the one thing in the current S2.2 (actually in S2.2.3) that I think is > worth keeping is a mention of fixed and mobile. I think this could be > handled by adding to the Intro (S1) something like: "Measurements will ta= ke > place on both fixed and mobile access" > > Happy to add a new chapter 4, but comments about some of your sub use cas= es > > 4.1 Promoting competition through transparency > >>For competition to successfully discipline operators' behaviour in the > interests of their customers, end users must be fully aware of the > characteristics of the ISPs' access offers. > "must be fully aware" over-states things. > > >>The system can be operated by a regulator or a measurement provider. > add "on behalf of the regulator" > > >> However, the quality of an Internet access service is also determined > by the connectivity to the rest of the Internet. Therefore test > configurations which measure beyond the ISP's own network may also be > relevant in some cases. > true. perhaps worth mentioning that the ISPs can influence this to some > extent - how well they're connected to the Internet, how they cache popul= ar > content etc. > > 4.2 Promoting end user empowerment > i think this can be deleted. this is the end user use case, not the > regulator use case. > > 4.3 Monitoring overall market development good for me > > 4.4 Detecting degradation of the Internet access > 4.5 Detecting degradation of specific applications > > I don't like these sections about testing for whether the ISP is being ne= t > neutral. > I think it would be very hard to design tests. > In part this is because net neutrality is more of a political concept tha= n > technical one (in my view) > > i tihnk some of the stuff in S4.4 is really part of 4.3, as it's a market > development issue (the internet has got slower, and doesnt meet the polic= y > objective of Superfast broadband by Year 2015 - or whatever)- > > best wishes > phil > > ________________________________________ > From: lmap-bounces@ietf.org [lmap-bounces@ietf.org] On Behalf Of > S=F8rensen, Frode [frode.sorensen@npt.no] > Sent: 30 October 2013 14:55 > To: lmap@ietf.org > Subject: [lmap] LMAP regulator use case > > Marc, Phil, Trevor, all, > > I have prepared a proposed description of a regulator use case for the > LMAP use case ID. I noticed the "advertisement" for further input from > regulators for the use case a couple of weeks ago. Finally I have managed > to find sufficient time to contribute to the ID, supported by a couple of > "regulator friends" from the Net Neutrality Working Group of the BEREC > organization, representing European national regulators. The proposal > consists of two parts; a modification of section 2.2 "Regulators", plus a > new chapter after chapter 3 "Details of ISP use case", titled "Details of > regulator use case". > > Thanks, > Frode > > > Proposed update: > > 2.2 Regulators > ------------------- > Regulators in jurisdictions around the world are responding to consumers' > adoption of Internet access services for traditional telecommunications a= nd > media services by promoting competition among providers of electronic > communications, to ensure that users derive maximum benefit in terms of > choice, price, and quality. > > Some jurisdictions have responded to a need for greater information about > Internet access service performance in the development of regulatory > policies and approaches for broadband technologies by developing > large-scale measurement programs. Programs such as the U.S. Federal > Communications Commission's Measuring Broadband America, European > Commission's Quality of Broadband Services in the EU reports and a growin= g > list of other programs employ a diverse set of operational and technical > approaches to gathering data to perform analysis and reporting on diverse > aspects of broadband performance. > > While each jurisdiction responds to distinct consumer, industry, and > regulatory concerns, much commonality exists in the need to produce > datasets that are able to compare multiple Internet access service > providers, diverse technical solutions, geographic and regional > distributions, and marketed and provisioned levels and combinations of > broadband Internet access services. In some jurisdictions, the role of > measuring is provided by a measurement provider. > > Measurement providers measure network performance from users towards > multiple content and application providers, included dedicated test > measurement servers, to show a performance of the actual Internet access > service provided by different ISPs. Users need to know the performance th= at > are achieving from their own ISP. In addition, they need to know the > performance of other ISPs of same location as background information for > selecting their ISP. Measurement providers will provide measurement resul= ts > with associated measurement methods and measurement metrics. > > _______________________________________________ > lmap mailing list > lmap@ietf.org > https://www.ietf.org/mailman/listinfo/lmap > --f46d0438907d10212b04ea7c889f Content-Type: text/html; charset=windows-1252 Content-Transfer-Encoding: quoted-printable
I would propose adding in section 2 or 4 some text de= scribing a regulators need to ensure that policies intended to protect the = privacy interests of a potential tester can be enforced using features avai= lable in the system. =A0Recognizing that=A0http://www.ietf.org/id/draft-folks-lma= p-framework-00.txt and in the lists there's a view that a single en= tity will be overseeing a large scale measurement, there is a clear need fo= r a test infrastructure to include tools to enforce the privacy policies of= the managing entity and coordinate collaborative activity. =A0A desire to = protect the privacy of testers contributing to our mobile measurement effor= t lead our work on the project, and our decision to adopt changes to our fi= xed technical infrastructure and data model to support a totally anonymous = collection at the MA and communications between controllers, and through th= e collectors and the test results collected. =A0I understand there has been= discussion on this point in the past and we had included this in the Secti= on 7 Security Considerations of our original I-D. =A0In any event I think t= he regulator use case benefits from calling the point out specifically.

Proposed addition in Section 2.2:=
::=A0Regul= ators role in the development and enforcement of broadband Internet access = service policies also require that the measurement approaches meet a high l= evel of verifiability, accuracy and provider-independence to support valid = and meaningful comparisons of Internet access service performance. =A0=A0Regulators also require that policies= intended to protect the privacy interests of potential testers can be enfo= rced using features available in the system. =A0


Naturally any use case is going to = have some variation among the specific practices of the parties. =A0On Klau= s and Frode's comments (and excellent draft contributions thanks!) I wo= uld chime in to add that "Transparency" is a fundamental feature = of the rules intended to protect the Open Internet (or network neutrality i= n other parlance). =A0Our rule is intended to enable "consumers to mak= e informed choices regarding use of such services" but also "for = content, application, service, and device providers to develop, market, and= maintain Internet offerings."=A047 CFR 8.3 - Transparency.

Whether protecting consumers is moved I think you= find the thought and regulatory use case solidly grounded in the US approa= ch as well. =A0I would also suggest adding some text to emphasize that Tran= sparency is also important to enabling development in the ecosystem.

Proposed addition in Section 4.1:
:: End = users need effective transparency to be able to make informed choices throu= ghout the different stages of their relationship with ISPs, when selecting = Internet access service offers, and when considering switching service offe= r within an ISP or to an alternative ISP. =A0Transparency in Internet perfo= rmance information is also important to content, application, service, and = device providers' to ensure they are able to develop, market, and maint= ain Internet offerings. =A0Quality information about service offering can b= e provided based on performance measurements of the services provided in th= e market.


On Klaus' comments on what in US jur= isdictions might be understand as "blocking" or "discriminat= ory" conduct, I'd add that US law includes similar provisions. 47 = C.F.R.=A7=A7 8.5, 8.7. =A0Recognizing that these topics including "Qua= lity of Service" are evolving and differ in many respects across juris= dictions, a high level description focused on the value of performance info= rmation to regulators in this regard may be helpful in the section.

I would also breach another topic for future discussion= on assessing quality on the test network itself. =A0To maintain comparabil= ity and quality of the collected data, the architecture itself must be moni= tored with auditing of whether the infrastructure is functioning correctly,= e.g. server health and connectivity. MA side is much easier to keep a hand= le on with special purpose probes (such as with our fixed "whitebox&qu= ot; approach) but server side checks on and between controllers is also a v= ery important issue to keep an eye on. =A0I don't have a specific propo= sal how to address this in an I-D right now but I'd note it as somethin= g any large scale measurement manager will need to address.
--
JamesMil= lerEsquire@Gmail.com

"Wovon man nicht spreche= n kann, dar=FCber mu=DF man schweigen."
(What we cannot spea= k about we must pass over in silence. =A0)
Ludwig Josef Johann Wittgenstein


On Wed, Nov 6, 2013 at 12:19 AM, Niemine= n Klaus <Klaus.Nieminen@ficora.fi> wrote:
Hi Philip, Frode and all,

As one of the =A0regulator friends that Frode mentioned, I would like to te= ll how we see these issues in Finland as I'm working for the Finnish te= lco regulator.

4.2 Promoting end user empowerment

I would like to see this chapter to be included. As said publishing t= ransparent and comparable information is important but so it's also end= user's ability to verify the performance of the Internet access servic= e provided, comparing the measurement results achieved with the information= given in the contract. If the contracted service quality level is not met,= end users can use these results when complaining to the ISP to achieve imp= rovement and compensation.

If you want to include it to end user requirements it is ok for me to refer= ence it from the regulator part. However I do not want it to be totally del= eted as it is also our requirement not only end user one.

4.4 Detecting degradation of the Internet access
4.5 Detecting degradation of specific applications

Yes I know that this is a hard problem how to measure NN. However it = is a clear regulatory requirement at least in the EU. See Directive 2009/13= 6/EC http://eur-lex.europa.eu/LexU= riServ/LexUriServ.do?uri=3DOJ:L:2009:337:0011:01:EN:HTML

According to Art 22(3), =93in order to prevent the degradation of service a= nd the hindering or slowing down of traffic over networks, Member States sh= all ensure that national regulatory authorities are able to set minimum qua= lity of service requirements on an undertaking or undertakings providing pu= blic communications networks.=94 The reach of this regulatory tool has been= extensively examined by BEREC, which sees it as the third and last step of= the regulatory approach to net neutrality breaches

So we do need tools for this purpose as Frode has described.

best regards,

- Klaus -
___________________________________________
KLAUS NIEMINEN
Communications Network Specialist
Finnish Communications Regulatory Authority
P.O. Box 313
FIN-00181 HELSINKI, FINLAND
tel.: +358= 29 5390 528
fax: +358 9 = 6966 873
e-mail: klaus.nieminen@ficora.f= i
www: http://www.ficora.f= i

-----Original Message-----
From: philip.eardley@bt.com [m= ailto:philip.eardley@bt.com] Sent: 6. marraskuuta 2013 4:42
To: frode.sorensen@npt.no; lmap@ietf.org
Subject: Re: [lmap] LMAP regulator use case

Frode
thanks very much for the comments - and it is excellent to have detailed te= xt suggestions.

2.2
I think i'm basically OK with this replacing the current S2.2 (includin= g the subsections) I suggest that it should mention comparability in the fi= nal para (as well as in the penultimate one) - personally I think this is t= he most important characteristic from a regulator's point of view, so t= hat it's fair to compare measurements they make across the different IS= Ps.
the one thing in the current S2.2 (actually in S2.2.3) that I think is wort= h keeping is a mention of fixed and mobile. I think this could be handled b= y adding to the Intro (S1) something like: "Measurements will take pla= ce on both fixed and mobile access"

Happy to add a new chapter 4, but comments about some of your sub use cases=

4.1 Promoting competition through transparency
>>For competition to successfully discipline operators' behaviour= in the interests of their customers, end users must be fully aware of the = characteristics of the ISPs' access offers.
"must be fully aware" over-states things.

>>The system can be operated by a regulator or a measurement provider= .
add "on behalf of the regulator"

>> =A0However, the quality of an Internet access service is also dete= rmined by the connectivity to the rest of the Internet. Therefore test conf= igurations which measure beyond the ISP's own network may also be relev= ant in some cases.
true. perhaps worth mentioning that the ISPs can influence this to some ext= ent - how well they're connected to the Internet, how they cache popula= r content etc.

4.2 Promoting end user empowerment
i think this can be deleted. this is the end user use case, not the regulat= or use case.

4.3 Monitoring overall market development good for me

4.4 Detecting degradation of the Internet access
4.5 Detecting degradation of specific applications

I don't like these sections about testing for whether the ISP is being = net neutral.
I think it would be very hard to design tests.
In part this is because net neutrality is more of a political concept than = technical one (in my view)

i tihnk some of the stuff in S4.4 is really part of 4.3, as it's a mark= et development issue (the internet has got slower, and doesnt meet the poli= cy objective of Superfast broadband by Year 2015 - or whatever)-

best wishes
phil

________________________________________
From: lmap-bounces@ietf.org [<= a href=3D"mailto:lmap-bounces@ietf.org">lmap-bounces@ietf.org] On Behal= f Of S=F8rensen, Frode [frode.sore= nsen@npt.no]
Sent: 30 October 2013 14:55
To: lmap@ietf.org
Subject: [lmap] LMAP regulator use case

Marc, Phil, Trevor, all,

I have prepared a proposed description of a regulator use case for the LMAP= use case ID. I noticed the "advertisement" for further input fro= m regulators for the use case a couple of weeks ago. Finally I have managed= to find sufficient time to contribute to the ID, supported by a couple of = "regulator friends" from the Net Neutrality Working Group of the = BEREC organization, representing European national regulators. The proposal= consists of two parts; a modification of section 2.2 "Regulators"= ;, plus a new chapter after chapter 3 "Details of ISP use case", = titled "Details of regulator use case".

Thanks,
Frode


Proposed update:

2.2 Regulators
-------------------
Regulators in jurisdictions around the world are responding to consumers= 9; adoption of Internet access services for traditional telecommunications = and media services by promoting competition among providers of electronic c= ommunications, to ensure that users derive maximum benefit in terms of choi= ce, price, and quality.

Some jurisdictions have responded to a need for greater information about I= nternet access service performance in the development of regulatory policie= s and approaches for broadband technologies by developing large-scale measu= rement programs. Programs such as the U.S. Federal Communications Commissio= n's Measuring Broadband America, European Commission's Quality of B= roadband Services in the EU reports and a growing list of other programs em= ploy a diverse set of operational and technical approaches to gathering dat= a to perform analysis and reporting on diverse aspects of broadband perform= ance.

While each jurisdiction responds to distinct consumer, industry, and regula= tory concerns, much commonality exists in the need to produce datasets that= are able to compare multiple Internet access service providers, diverse te= chnical solutions, geographic and regional distributions, and marketed and = provisioned levels and combinations of broadband Internet access services. = In some jurisdictions, the role of measuring is provided by a measurement p= rovider.

Measurement providers measure network performance from users towards multip= le content and application providers, included dedicated test measurement s= ervers, to show a performance of the actual Internet access service provide= d by different ISPs. Users need to know the performance that are achieving = from their own ISP. In addition, they need to know the performance of other= ISPs of same location as background information for selecting their ISP. M= easurement providers will provide measurement results with associated measu= rement methods and measurement metrics.

_____________________________= __________________
lmap mailing list
lmap@ietf.org
ht= tps://www.ietf.org/mailman/listinfo/lmap

--f46d0438907d10212b04ea7c889f-- From frode.sorensen@npt.no Tue Nov 5 23:18:29 2013 Return-Path: X-Original-To: lmap@ietfa.amsl.com Delivered-To: lmap@ietfa.amsl.com Received: from localhost (localhost [127.0.0.1]) by ietfa.amsl.com (Postfix) with ESMTP id ACF3111E80F8 for ; Tue, 5 Nov 2013 23:18:29 -0800 (PST) X-Virus-Scanned: amavisd-new at amsl.com X-Spam-Flag: NO X-Spam-Score: -4.365 X-Spam-Level: X-Spam-Status: No, score=-4.365 tagged_above=-999 required=5 tests=[AWL=1.932, BAYES_00=-2.599, HTML_MESSAGE=0.001, MIME_8BIT_HEADER=0.3, RCVD_IN_DNSWL_MED=-4, UNPARSEABLE_RELAY=0.001] Received: from mail.ietf.org ([12.22.58.30]) by localhost (ietfa.amsl.com [127.0.0.1]) (amavisd-new, port 10024) with ESMTP id fWiqy8oEdz6F for ; Tue, 5 Nov 2013 23:18:20 -0800 (PST) Received: from mail1.bemta14.messagelabs.com (mail1.bemta14.messagelabs.com [193.109.254.117]) by ietfa.amsl.com (Postfix) with ESMTP id EB32911E80E0 for ; Tue, 5 Nov 2013 23:18:19 -0800 (PST) Received: from [193.109.254.147:46191] by server-13.bemta-14.messagelabs.com id 90/EE-17468-A3DE9725; Wed, 06 Nov 2013 07:18:18 +0000 X-Env-Sender: frode.sorensen@npt.no X-Msg-Ref: server-7.tower-27.messagelabs.com!1383722297!954210!1 X-Originating-IP: [213.225.64.154] X-StarScan-Received: X-StarScan-Version: 6.9.13; banners=-,-,- X-VirusChecked: Checked Received: (qmail 29670 invoked from network); 6 Nov 2013 07:18:17 -0000 Received: from unknown (HELO EXCAS.npta.no) (213.225.64.154) by server-7.tower-27.messagelabs.com with AES128-SHA encrypted SMTP; 6 Nov 2013 07:18:17 -0000 Received: from EXMBX01.npta.no ([10.10.2.97]) by EXCAS.npta.no ([fe80::b007:5474:dccd:c4dd%11]) with mapi id 14.02.0347.000; Wed, 6 Nov 2013 08:18:16 +0100 From: =?iso-8859-1?Q?S=F8rensen=2C_Frode?= To: "lmap@ietf.org" Thread-Topic: [lmap] LMAP regulator use case Thread-Index: Ac7av/gPWUtKN1yKQ0+keN6+1dM0dw== Date: Wed, 6 Nov 2013 07:18:15 +0000 Message-ID: <793D91975B99224DA9777562FF192808A274A652@exmbx01> Accept-Language: nb-NO, en-US Content-Language: nb-NO X-MS-Has-Attach: X-MS-TNEF-Correlator: x-originating-ip: [10.10.8.40] Content-Type: multipart/alternative; boundary="_000_793D91975B99224DA9777562FF192808A274A652exmbx01_" MIME-Version: 1.0 Cc: Nieminen Klaus , "philip.eardley@bt.com" , Marc Linsner , James Miller , Henning Schulzrinne Subject: Re: [lmap] LMAP regulator use case X-BeenThere: lmap@ietf.org X-Mailman-Version: 2.1.12 Precedence: list List-Id: Large Scale Measurement of Access network Performance List-Unsubscribe: , List-Archive: List-Post: List-Help: List-Subscribe: , X-List-Received-Date: Wed, 06 Nov 2013 07:18:29 -0000 --_000_793D91975B99224DA9777562FF192808A274A652exmbx01_ Content-Type: text/plain; charset="iso-8859-1" Content-Transfer-Encoding: quoted-printable Dear all, Thanks for your feedback on the regulator use cases, Phil. I support many o= f your suggestions for improving the text of the use cases, in particular, = mentioning comparability in current section 2.2, but also mentioning fixed = and mobile. In my view, sub cases 4.2, 4.4 and 4.5 are relevant, but also important. I = agree with Klaus' and James' comments, thank you both. I support the additi= ons proposed by James. I would also like to add a couple of other aspects regarding these sub case= s: 4.2 Promoting end user empowerment I agree with Phil that this is also aka end user use case. But the regulato= r is an important stakeholder in the use case. Klaus mentions some regulato= ry aspects. In addition, it is a fact that in some jurisdictions the regula= tor, or a third party on behalf of the regulator, operates a speed meter fo= r example. (However, I do not think Ofcom (UK) does this.) An important rea= son for the regulator's engagement is to achieve comparability between oper= ators. 4.4 Detecting degradation of the Internet access 4.5 Detecting degradation of specific applications These use cases already exit as regulator use cases externally to LMAP. For= example, the "BEREC Guidelines for quality of service in the scope of net = neutrality" available at http://berec.europa.eu/eng/document_register/subject_matter/berec/regulator= y_best_practices/guidelines/?doc=3D1101 describes the background for these use cases in chapters 4 and 5 respective= ly: 4. Degradation of Internet access service as a whole 5. Degradation of Internet access service with regard to individual applica= tions Use case 4.4 is probably also relevant for FCC since they in their press re= lease regarding the open Internet report & order say for example "We will c= losely monitor the robustness and affordability of broadband Internet acces= s services, with a particular focus on any signs that specialized services = are in any way retarding the growth of or constricting capacity available f= or broadband Internet access service." It is also interesting that there already exist measurement tools that supp= ort use case 4.5, notable Glasnost which is available at http://www.measure= mentlab.net/tools/glasnost. However, I agree with Phil that these two use cases are challenging to perf= orm. All the more, large scale measurements are needed to perform them in a= n accurate and verifiable way. Therefore, I think these use cases should be= kept. Sorry that I am not able to attend this IETF meeting, but I am looking forw= ard to join you at the next meeting, Frode Fra: James Miller [mailto:jamesmilleresquire@gmail.com] Sendt: 6. november 2013 07:47 Til: Nieminen Klaus Kopi: philip.eardley@bt.com; S=F8rensen, Frode; lmap@ietf.org; Marc Linsner= ; Henning Schulzrinne Emne: Re: [lmap] LMAP regulator use case I would propose adding in section 2 or 4 some text describing a regulators = need to ensure that policies intended to protect the privacy interests of a= potential tester can be enforced using features available in the system. = Recognizing that http://www.ietf.org/id/draft-folks-lmap-framework-00.txt a= nd in the lists there's a view that a single entity will be overseeing a la= rge scale measurement, there is a clear need for a test infrastructure to i= nclude tools to enforce the privacy policies of the managing entity and coo= rdinate collaborative activity. A desire to protect the privacy of testers= contributing to our mobile measurement effort lead our work on the project= , and our decision to adopt changes to our fixed technical infrastructure a= nd data model to support a totally anonymous collection at the MA and commu= nications between controllers, and through the collectors and the test resu= lts collected. I understand there has been discussion on this point in the= past and we had included this in the Section 7 Security Considerations of = our original I-D. In any event I think the regulator use case benefits fro= m calling the point out specifically. http://tools.ietf.org/id/draft-schulzrinne-lmap-requirements-00.txt Proposed addition in Section 2.2: :: Regulators role in the development and enforcement of broadband Internet= access service policies also require that the measurement approaches meet = a high level of verifiability, accuracy and provider-independence to suppor= t valid and meaningful comparisons of Internet access service performance. = Regulators also require that policies intended to protect the privacy int= erests of potential testers can be enforced using features available in the= system. Naturally any use case is going to have some variation among the specific p= ractices of the parties. On Klaus and Frode's comments (and excellent draf= t contributions thanks!) I would chime in to add that "Transparency" is a f= undamental feature of the rules intended to protect the Open Internet (or n= etwork neutrality in other parlance). Our rule is intended to enable "cons= umers to make informed choices regarding use of such services" but also "fo= r content, application, service, and device providers to develop, market, a= nd maintain Internet offerings." 47 CFR 8.3 - Transparency. Whether protecting consumers is moved I think you find the thought and regu= latory use case solidly grounded in the US approach as well. I would also = suggest adding some text to emphasize that Transparency is also important t= o enabling development in the ecosystem. Proposed addition in Section 4.1: :: End users need effective transparency to be able to make informed choice= s throughout the different stages of their relationship with ISPs, when sel= ecting Internet access service offers, and when considering switching servi= ce offer within an ISP or to an alternative ISP. Transparency in Internet = performance information is also important to content, application, service,= and device providers' to ensure they are able to develop, market, and main= tain Internet offerings. Quality information about service offering can be= provided based on performance measurements of the services provided in the= market. On Klaus' comments on what in US jurisdictions might be understand as "bloc= king" or "discriminatory" conduct, I'd add that US law includes similar pro= visions. 47 C.F.R.=A7=A7 8.5, 8.7. Recognizing that these topics including= "Quality of Service" are evolving and differ in many respects across juris= dictions, a high level description focused on the value of performance info= rmation to regulators in this regard may be helpful in the section. I would also breach another topic for future discussion on assessing qualit= y on the test network itself. To maintain comparability and quality of the= collected data, the architecture itself must be monitored with auditing of= whether the infrastructure is functioning correctly, e.g. server health an= d connectivity. MA side is much easier to keep a handle on with special pur= pose probes (such as with our fixed "whitebox" approach) but server side ch= ecks on and between controllers is also a very important issue to keep an e= ye on. I don't have a specific proposal how to address this in an I-D righ= t now but I'd note it as something any large scale measurement manager will= need to address. -- JamesMillerEsquire@Gmail.com "Wovon man nicht sprechen kann, dar=FCber mu=DF man schweigen." (What we cannot speak about we must pass over in silence. ) Ludwig Josef Johann Wittgenstein On Wed, Nov 6, 2013 at 12:19 AM, Nieminen Klaus > wrote: Hi Philip, Frode and all, As one of the regulator friends that Frode mentioned, I would like to tell= how we see these issues in Finland as I'm working for the Finnish telco re= gulator. 4.2 Promoting end user empowerment I would like to see this chapter to be included. As said publishing transpa= rent and comparable information is important but so it's also end user's ab= ility to verify the performance of the Internet access service provided, co= mparing the measurement results achieved with the information given in the = contract. If the contracted service quality level is not met, end users can= use these results when complaining to the ISP to achieve improvement and c= ompensation. If you want to include it to end user requirements it is ok for me to refer= ence it from the regulator part. However I do not want it to be totally del= eted as it is also our requirement not only end user one. 4.4 Detecting degradation of the Internet access 4.5 Detecting degradation of specific applications Yes I know that this is a hard problem how to measure NN. However it is a c= lear regulatory requirement at least in the EU. See Directive 2009/136/EC h= ttp://eur-lex.europa.eu/LexUriServ/LexUriServ.do?uri=3DOJ:L:2009:337:0011:0= 1:EN:HTML According to Art 22(3), "in order to prevent the degradation of service and= the hindering or slowing down of traffic over networks, Member States shal= l ensure that national regulatory authorities are able to set minimum quali= ty of service requirements on an undertaking or undertakings providing publ= ic communications networks." The reach of this regulatory tool has been ext= ensively examined by BEREC, which sees it as the third and last step of the= regulatory approach to net neutrality breaches So we do need tools for this purpose as Frode has described. best regards, - Klaus - ___________________________________________ KLAUS NIEMINEN Communications Network Specialist Finnish Communications Regulatory Authority P.O. Box 313 FIN-00181 HELSINKI, FINLAND tel.: +358 29 5390 528 fax: +358 9 6966 873 e-mail: klaus.nieminen@ficora.fi www: http://www.ficora.fi -----Original Message----- From: philip.eardley@bt.com [mailto:philip.ea= rdley@bt.com] Sent: 6. marraskuuta 2013 4:42 To: frode.sorensen@npt.no; lmap@ietf.org Subject: Re: [lmap] LMAP regulator use case Frode thanks very much for the comments - and it is excellent to have detailed te= xt suggestions. 2.2 I think i'm basically OK with this replacing the current S2.2 (including th= e subsections) I suggest that it should mention comparability in the final = para (as well as in the penultimate one) - personally I think this is the m= ost important characteristic from a regulator's point of view, so that it's= fair to compare measurements they make across the different ISPs. the one thing in the current S2.2 (actually in S2.2.3) that I think is wort= h keeping is a mention of fixed and mobile. I think this could be handled b= y adding to the Intro (S1) something like: "Measurements will take place on= both fixed and mobile access" Happy to add a new chapter 4, but comments about some of your sub use cases 4.1 Promoting competition through transparency >>For competition to successfully discipline operators' behaviour in the in= terests of their customers, end users must be fully aware of the characteri= stics of the ISPs' access offers. "must be fully aware" over-states things. >>The system can be operated by a regulator or a measurement provider. add "on behalf of the regulator" >> However, the quality of an Internet access service is also determined b= y the connectivity to the rest of the Internet. Therefore test configuratio= ns which measure beyond the ISP's own network may also be relevant in some = cases. true. perhaps worth mentioning that the ISPs can influence this to some ext= ent - how well they're connected to the Internet, how they cache popular co= ntent etc. 4.2 Promoting end user empowerment i think this can be deleted. this is the end user use case, not the regulat= or use case. 4.3 Monitoring overall market development good for me 4.4 Detecting degradation of the Internet access 4.5 Detecting degradation of specific applications I don't like these sections about testing for whether the ISP is being net = neutral. I think it would be very hard to design tests. In part this is because net neutrality is more of a political concept than = technical one (in my view) i tihnk some of the stuff in S4.4 is really part of 4.3, as it's a market d= evelopment issue (the internet has got slower, and doesnt meet the policy o= bjective of Superfast broadband by Year 2015 - or whatever)- best wishes phil ________________________________________ From: lmap-bounces@ietf.org [lmap-bounces@iet= f.org] On Behalf Of S=F8rensen, Frode [frode.= sorensen@npt.no] Sent: 30 October 2013 14:55 To: lmap@ietf.org Subject: [lmap] LMAP regulator use case Marc, Phil, Trevor, all, I have prepared a proposed description of a regulator use case for the LMAP= use case ID. I noticed the "advertisement" for further input from regulato= rs for the use case a couple of weeks ago. Finally I have managed to find s= ufficient time to contribute to the ID, supported by a couple of "regulator= friends" from the Net Neutrality Working Group of the BEREC organization, = representing European national regulators. The proposal consists of two par= ts; a modification of section 2.2 "Regulators", plus a new chapter after ch= apter 3 "Details of ISP use case", titled "Details of regulator use case". Thanks, Frode Proposed update: 2.2 Regulators ------------------- Regulators in jurisdictions around the world are responding to consumers' a= doption of Internet access services for traditional telecommunications and = media services by promoting competition among providers of electronic commu= nications, to ensure that users derive maximum benefit in terms of choice, = price, and quality. Some jurisdictions have responded to a need for greater information about I= nternet access service performance in the development of regulatory policie= s and approaches for broadband technologies by developing large-scale measu= rement programs. Programs such as the U.S. Federal Communications Commissio= n's Measuring Broadband America, European Commission's Quality of Broadband= Services in the EU reports and a growing list of other programs employ a d= iverse set of operational and technical approaches to gathering data to per= form analysis and reporting on diverse aspects of broadband performance. While each jurisdiction responds to distinct consumer, industry, and regula= tory concerns, much commonality exists in the need to produce datasets that= are able to compare multiple Internet access service providers, diverse te= chnical solutions, geographic and regional distributions, and marketed and = provisioned levels and combinations of broadband Internet access services. = In some jurisdictions, the role of measuring is provided by a measurement p= rovider. Measurement providers measure network performance from users towards multip= le content and application providers, included dedicated test measurement s= ervers, to show a performance of the actual Internet access service provide= d by different ISPs. Users need to know the performance that are achieving = from their own ISP. In addition, they need to know the performance of other= ISPs of same location as background information for selecting their ISP. M= easurement providers will provide measurement results with associated measu= rement methods and measurement metrics. _______________________________________________ lmap mailing list lmap@ietf.org https://www.ietf.org/mailman/listinfo/lmap --_000_793D91975B99224DA9777562FF192808A274A652exmbx01_ Content-Type: text/html; charset="iso-8859-1" Content-Transfer-Encoding: quoted-printable

Dear all,<= o:p>

 = ;

Thanks for= your feedback on the regulator use cases, Phil. I support many of your sug= gestions for improving the text of the use cases, in particular, mentioning comparability in current section 2.2, but also mentioning fixed= and mobile.

 = ;

In my view= , sub cases 4.2, 4.4 and 4.5 are relevant, but also important. I agree with= Klaus’ and James’ comments, thank you both. I support the additions proposed by James.

 = ;

I would al= so like to add a couple of other aspects regarding these sub cases:

 = ;

4.2 Promot= ing end user empowerment

I agree wi= th Phil that this is also aka end user use case. But the regulator is an im= portant stakeholder in the use case. Klaus mentions some regulatory aspects. In addition, it is a fact that in some jurisdictions the regulato= r, or a third party on behalf of the regulator, operates a speed meter for = example. (However, I do not think Ofcom (UK) does this.) An important reaso= n for the regulator’s engagement is to achieve comparability between operators.

 = ;

4.4 Detect= ing degradation of the Internet access

4.5 Detect= ing degradation of specific applications

 = ;

These use = cases already exit as regulator use cases externally to LMAP. For example, = the “BEREC Guidelines for quality of service in the scope of net neutrality” available at

http://ber= ec.europa.eu/eng/document_register/subject_matter/berec/regulatory_best_pra= ctices/guidelines/?doc=3D1101

describes = the background for these use cases in chapters 4 and 5 respectively:

4. Degrada= tion of Internet access service as a whole

5. Degrada= tion of Internet access service with regard to individual applications=

 = ;

Use case 4= .4 is probably also relevant for FCC since they in their press release rega= rding the open Internet report & order say for example “We will closely monitor the robustness and affordability of broadband Interne= t access services, with a particular focus on any signs that specialized se= rvices are in any way retarding the growth of or constricting capacity avai= lable for broadband Internet access service.”

 = ;

It is also= interesting that there already exist measurement tools that support use ca= se 4.5, notable Glasnost which is available at http://www.measurementlab.ne= t/tools/glasnost.

 = ;

However, I= agree with Phil that these two use cases are challenging to perform. All t= he more, large scale measurements are needed to perform them in an accurate and verifiable way. Therefore, I think these use cases shou= ld be kept.

 = ;

Sorry that= I am not able to attend this IETF meeting, but I am looking forward to joi= n you at the next meeting,

Frode

 <= /p>

 <= /p>

Fra: James Mil= ler [mailto:jamesmilleresquire@gmail.com]
Sendt: 6. november 2013 07:47
Til: Nieminen Klaus
Kopi: philip.eardley@bt.com; S=F8rensen, Frode; lmap@ietf.org; Marc = Linsner; Henning Schulzrinne
Emne: Re: [lmap] LMAP regulator use case

 

I would propose adding in section 2 or 4 some text d= escribing a regulators need to ensure that policies intended to protect the= privacy interests of a potential tester can be enforced using features ava= ilable in the system.  Recognizing that http://www.ietf.org/id/draft-folks-lmap-framework-00.txt and in th= e lists there's a view that a single entity will be overseeing a large scal= e measurement, there is a clear need for a test infrastructure to include tools to enforce the privacy policies of = the managing entity and coordinate collaborative activity.  A desire t= o protect the privacy of testers contributing to our mobile measurement eff= ort lead our work on the project, and our decision to adopt changes to our fixed technical infrastructure and da= ta model to support a totally anonymous collection at the MA and communicat= ions between controllers, and through the collectors and the test results c= ollected.  I understand there has been discussion on this point in the past and we had included this in the = Section 7 Security Considerations of our original I-D.  In any event I= think the regulator use case benefits from calling the point out specifica= lly.

 

Proposed addition in Section 2.2:

:: Regulators role in the developmen= t and enforcement of broadband Internet access service policies also requir= e that the measurement approaches meet a high level of verifiability, accuracy and provider-independence to support valid and meaningful compari= sons of Internet access service performance.   Regulators also r= equire that policies intended to protect the privacy interests of potential testers can be enforced using features available in the system. =  

 

 

Naturally any use case is going to have some variati= on among the specific practices of the parties.  On Klaus and Frode's = comments (and excellent draft contributions thanks!) I would chime in to ad= d that "Transparency" is a fundamental feature of the rules intended to protect the Open Internet (or network neu= trality in other parlance).  Our rule is intended to enable "cons= umers to make informed choices regarding use of such services" but als= o "for content, application, service, and device providers to develop, market, and maintain Internet offerings." = 47 CFR 8.3 - Transparency.

 

Whether protecting consumers is moved I think you fi= nd the thought and regulatory use case solidly grounded in the US approach = as well.  I would also suggest adding some text to emphasize that Tran= sparency is also important to enabling development in the ecosystem.

 

Proposed addition in Section 4.1:

:: End users need effective transparency to be able = to make informed choices throughout the different stages of their relations= hip with ISPs, when selecting Internet access service offers, and when cons= idering switching service offer within an ISP or to an alternative ISP.  Transparency in Internet performanc= e information is also important to content, application, service, and devic= e providers' to ensure they are able to develop, market, and maintain Inter= net offerings.  Quality information about service offering can be provided based on performance measurements of the = services provided in the market.

 

 

On Klaus' comments on what in US jurisdictions might= be understand as "blocking" or "discriminatory" conduc= t, I'd add that US law includes similar provisions. 47 C.F.R.=A7=A7 8.5, 8.= 7.  Recognizing that these topics including "Quality of Service&q= uot; are evolving and differ in many respects across jurisdictions, a high leve= l description focused on the value of performance information to regulators= in this regard may be helpful in the section.

 

I would also breach another topic for future discuss= ion on assessing quality on the test network itself.  To maintain comp= arability and quality of the collected data, the architecture itself must b= e monitored with auditing of whether the infrastructure is functioning correctly, e.g. server health and connectivi= ty. MA side is much easier to keep a handle on with special purpose probes = (such as with our fixed "whitebox" approach) but server side chec= ks on and between controllers is also a very important issue to keep an eye on.  I don't have a specific proposal = how to address this in an I-D right now but I'd note it as something any la= rge scale measurement manager will need to address.

--

 

"Wovon man nicht sprechen kann, dar=FCber mu=DF= man schweigen."

(What we cannot speak about we must pass over in sil= ence.  )

Ludwig Josef Johann Wittgenstein

 

On Wed, Nov 6, 2013 at 12:19 AM, Nieminen Klaus <= Klaus.Niemine= n@ficora.fi> wrote:

Hi Philip, Frode and all,

As one of the  regulator friends that Frode mentioned, I would like to= tell how we see these issues in Finland as I'm working for the Finnish tel= co regulator.


4.2 Promoting end user empowerment

I would like to see this chapter to be included. As = said publishing transparent and comparable information is important but so = it's also end user's ability to verify the performance of the Internet acce= ss service provided, comparing the measurement results achieved with the information given in the contract. I= f the contracted service quality level is not met, end users can use these = results when complaining to the ISP to achieve improvement and compensation= .

If you want to include it to end user requirements it is ok for me to refer= ence it from the regulator part. However I do not want it to be totally del= eted as it is also our requirement not only end user one.


4.4 Detecting degradation of the Internet access
4.5 Detecting degradation of specific applications

Yes I know that this is a hard problem how to measur= e NN. However it is a clear regulatory requirement at least in the EU. See = Directive 2009/136/EC http://eur-lex.europa.eu/LexUriServ/LexUriServ.do?uri=3DOJ:L:2009:337:0011:= 01:EN:HTML

According to Art 22(3), “in order to prevent the degradation of servi= ce and the hindering or slowing down of traffic over networks, Member State= s shall ensure that national regulatory authorities are able to set minimum= quality of service requirements on an undertaking or undertakings providing public communications networks.̶= 1; The reach of this regulatory tool has been extensively examined by BEREC= , which sees it as the third and last step of the regulatory approach to ne= t neutrality breaches

So we do need tools for this purpose as Frode has described.

best regards,

- Klaus -
___________________________________________
KLAUS NIEMINEN
Communications Network Specialist
Finnish Communications Regulatory Authority
P.O. Box 313
FIN-00181 HELSINKI, FINLAND
tel.: +358 29 5390 528
fax: +358 9 6966 873
e-mail: klaus.nieminen@ficora.f= i
www: http://www.ficora.f= i


-----Original Message-----
From: philip.eardley@bt.com [m= ailto:philip.eardley@bt.com] Sent: 6. marraskuuta 2013 4:42
To: frode.sorensen@npt.no; lmap@ietf.org
Subject: Re: [lmap] LMAP regulator use case

Frode
thanks very much for the comments - and it is excellent to have detailed te= xt suggestions.

2.2
I think i'm basically OK with this replacing the current S2.2 (including th= e subsections) I suggest that it should mention comparability in the final = para (as well as in the penultimate one) - personally I think this is the m= ost important characteristic from a regulator's point of view, so that it's fair to compare measurements the= y make across the different ISPs.
the one thing in the current S2.2 (actually in S2.2.3) that I think is wort= h keeping is a mention of fixed and mobile. I think this could be handled b= y adding to the Intro (S1) something like: "Measurements will take pla= ce on both fixed and mobile access"

Happy to add a new chapter 4, but comments about some of your sub use cases=

4.1 Promoting competition through transparency
>>For competition to successfully discipline operators' behaviour in = the interests of their customers, end users must be fully aware of the char= acteristics of the ISPs' access offers.
"must be fully aware" over-states things.

>>The system can be operated by a regulator or a measurement provider= .
add "on behalf of the regulator"

>>  However, the quality of an Internet access service is also d= etermined by the connectivity to the rest of the Internet. Therefore test c= onfigurations which measure beyond the ISP's own network may also be releva= nt in some cases.
true. perhaps worth mentioning that the ISPs can influence this to some ext= ent - how well they're connected to the Internet, how they cache popular co= ntent etc.

4.2 Promoting end user empowerment
i think this can be deleted. this is the end user use case, not the regulat= or use case.

4.3 Monitoring overall market development good for me

4.4 Detecting degradation of the Internet access
4.5 Detecting degradation of specific applications

I don't like these sections about testing for whether the ISP is being net = neutral.
I think it would be very hard to design tests.
In part this is because net neutrality is more of a political concept than = technical one (in my view)

i tihnk some of the stuff in S4.4 is really part of 4.3, as it's a market d= evelopment issue (the internet has got slower, and doesnt meet the policy o= bjective of Superfast broadband by Year 2015 - or whatever)-

best wishes
phil

________________________________________
From: lmap-bounces@ietf.org [<= a href=3D"mailto:lmap-bounces@ietf.org">lmap-bounces@ietf.org] On Behal= f Of S=F8rensen, Frode [frode.sore= nsen@npt.no]
Sent: 30 October 2013 14:55
To: lmap@ietf.org
Subject: [lmap] LMAP regulator use case

Marc, Phil, Trevor, all,

I have prepared a proposed description of a regulator use case for the LMAP= use case ID. I noticed the "advertisement" for further input fro= m regulators for the use case a couple of weeks ago. Finally I have managed= to find sufficient time to contribute to the ID, supported by a couple of "regulator friends" from the Ne= t Neutrality Working Group of the BEREC organization, representing European= national regulators. The proposal consists of two parts; a modification of= section 2.2 "Regulators", plus a new chapter after chapter 3 "Details of ISP use case", titled "Details = of regulator use case".

Thanks,
Frode


Proposed update:

2.2 Regulators
-------------------
Regulators in jurisdictions around the world are responding to consumers' a= doption of Internet access services for traditional telecommunications and = media services by promoting competition among providers of electronic commu= nications, to ensure that users derive maximum benefit in terms of choice, price, and quality.

Some jurisdictions have responded to a need for greater information about I= nternet access service performance in the development of regulatory policie= s and approaches for broadband technologies by developing large-scale measu= rement programs. Programs such as the U.S. Federal Communications Commission's Measuring Broadband America, = European Commission's Quality of Broadband Services in the EU reports and a= growing list of other programs employ a diverse set of operational and tec= hnical approaches to gathering data to perform analysis and reporting on diverse aspects of broadband performa= nce.

While each jurisdiction responds to distinct consumer, industry, and regula= tory concerns, much commonality exists in the need to produce datasets that= are able to compare multiple Internet access service providers, diverse te= chnical solutions, geographic and regional distributions, and marketed and provisioned levels and combinatio= ns of broadband Internet access services. In some jurisdictions, the role o= f measuring is provided by a measurement provider.

Measurement providers measure network performance from users towards multip= le content and application providers, included dedicated test measurement s= ervers, to show a performance of the actual Internet access service provide= d by different ISPs. Users need to know the performance that are achieving from their own ISP. In addition= , they need to know the performance of other ISPs of same location as backg= round information for selecting their ISP. Measurement providers will provi= de measurement results with associated measurement methods and measurement metrics.

_______________________________________________
lmap mailing list
lmap@ietf.org
ht= tps://www.ietf.org/mailman/listinfo/lmap

 

--_000_793D91975B99224DA9777562FF192808A274A652exmbx01_-- From timothy.carey@alcatel-lucent.com Wed Nov 6 04:05:30 2013 Return-Path: X-Original-To: lmap@ietfa.amsl.com Delivered-To: lmap@ietfa.amsl.com Received: from localhost (localhost [127.0.0.1]) by ietfa.amsl.com (Postfix) with ESMTP id 21ABB21E80CB for ; Wed, 6 Nov 2013 04:05:30 -0800 (PST) X-Virus-Scanned: amavisd-new at amsl.com X-Spam-Flag: NO X-Spam-Score: -10.271 X-Spam-Level: X-Spam-Status: No, score=-10.271 tagged_above=-999 required=5 tests=[AWL=0.327, BAYES_00=-2.599, HTML_MESSAGE=0.001, RCVD_IN_DNSWL_HI=-8] Received: from mail.ietf.org ([12.22.58.30]) by localhost (ietfa.amsl.com [127.0.0.1]) (amavisd-new, port 10024) with ESMTP id r21Rjk-h8R-T for ; Wed, 6 Nov 2013 04:05:23 -0800 (PST) Received: from ihemail4.lucent.com (ihemail4.lucent.com [135.245.0.39]) by ietfa.amsl.com (Postfix) with ESMTP id 72FB121E80B4 for ; Wed, 6 Nov 2013 04:05:23 -0800 (PST) Received: from us70tusmtp2.zam.alcatel-lucent.com (h135-5-2-64.lucent.com [135.5.2.64]) by ihemail4.lucent.com (8.13.8/IER-o) with ESMTP id rA6C5KcS008681 (version=TLSv1/SSLv3 cipher=DHE-RSA-AES256-SHA bits=256 verify=FAIL) for ; Wed, 6 Nov 2013 06:05:21 -0600 (CST) Received: from US70TWXCHHUB03.zam.alcatel-lucent.com (us70twxchhub03.zam.alcatel-lucent.com [135.5.2.35]) by us70tusmtp2.zam.alcatel-lucent.com (GMO) with ESMTP id rA6C5KIO005659 (version=TLSv1/SSLv3 cipher=AES128-SHA bits=128 verify=FAIL) for ; Wed, 6 Nov 2013 07:05:20 -0500 Received: from US70UWXCHMBA05.zam.alcatel-lucent.com ([169.254.10.30]) by US70TWXCHHUB03.zam.alcatel-lucent.com ([135.5.2.35]) with mapi id 14.02.0247.003; Wed, 6 Nov 2013 07:05:20 -0500 From: "Carey, Timothy (Timothy)" To: "lmap@ietf.org" Thread-Topic: Comments on draft-burrbridge-lmap-information-model-01 Thread-Index: Ac7a6HZ5T6s7aqiQTHydIbHLJBrc7A== Date: Wed, 6 Nov 2013 12:05:20 +0000 Message-ID: <9966516C6EB5FC4381E05BF80AA55F7720B35C@US70UWXCHMBA05.zam.alcatel-lucent.com> Accept-Language: en-US Content-Language: en-US X-MS-Has-Attach: X-MS-TNEF-Correlator: x-originating-ip: [135.5.27.18] Content-Type: multipart/alternative; boundary="_000_9966516C6EB5FC4381E05BF80AA55F7720B35CUS70UWXCHMBA05zam_" MIME-Version: 1.0 X-Scanned-By: MIMEDefang 2.57 on 135.245.2.39 Subject: [lmap] Comments on draft-burrbridge-lmap-information-model-01 X-BeenThere: lmap@ietf.org X-Mailman-Version: 2.1.12 Precedence: list List-Id: Large Scale Measurement of Access network Performance List-Unsubscribe: , List-Archive: List-Post: List-Help: List-Subscribe: , X-List-Received-Date: Wed, 06 Nov 2013 12:05:30 -0000 --_000_9966516C6EB5FC4381E05BF80AA55F7720B35CUS70UWXCHMBA05zam_ Content-Type: text/plain; charset="us-ascii" Content-Transfer-Encoding: quoted-printable Trevor, I was utilizing this draft for a submission into BBF Q4 meeting and had a c= ouple of comments and a question. 1) In section 2.4 The Report Channel Names description is the same as = the Measurement Task Configuration Names; probably a copy paste error 2) In section 2.2 The pre-configuration information is still types as = a MAC address; in the BBF we use a device identifier; could I suggest we ch= ange this to Device Identifier and a URN. 3) In section 2.4 the Measurement Task options now has a sub-attribute= of Interface Name - which is OK, but I wanted to know if this was the only= option allowed or their will be more opaque options along with this explic= it option. BTW - Just a comment; I did notice that the major change in the draft was a= n introduction of the channel concept; it does seem alittle overkill to hav= e separate Instruction, Logging and Status channels since the same MA is ta= lking to the same Controller. I guess we are weighing flexibility with comp= lexity; remember these MAs could be quite constrained... BR, Tim --_000_9966516C6EB5FC4381E05BF80AA55F7720B35CUS70UWXCHMBA05zam_ Content-Type: text/html; charset="us-ascii" Content-Transfer-Encoding: quoted-printable

Trevor,

 

I was utilizing this draft for a submission into BBF= Q4 meeting and had a couple of comments and a question.

 

1)      In section 2.4 The Report Channel Names description= is the same as the Measurement Task Configuration Names; probably a copy p= aste error

2)      In section 2.2 The pre-configuration information is= still types as a MAC address; in the BBF we use a device identifier; could= I suggest we change this to Device Identifier and a URN.

3)      In section 2.4 the Measurement Task options now has= a sub-attribute of Interface Name – which is OK, but I wanted to kno= w if this was the only option allowed or their will be more opaque options = along with this explicit option.

 

 

BTW – Just a comment; I did notice that the ma= jor change in the draft was an introduction of the channel concept; it does= seem alittle overkill to have separate Instruction, Logging and Status cha= nnels since the same MA is talking to the same Controller. I guess we are weighing flexibility with complexity; reme= mber these MAs could be quite constrained…

 

 

BR,

Tim

 

--_000_9966516C6EB5FC4381E05BF80AA55F7720B35CUS70UWXCHMBA05zam_-- From trevor.burbridge@bt.com Wed Nov 6 04:17:33 2013 Return-Path: X-Original-To: lmap@ietfa.amsl.com Delivered-To: lmap@ietfa.amsl.com Received: from localhost (localhost [127.0.0.1]) by ietfa.amsl.com (Postfix) with ESMTP id BD91A21E80AD for ; Wed, 6 Nov 2013 04:17:33 -0800 (PST) X-Virus-Scanned: amavisd-new at amsl.com X-Spam-Flag: NO X-Spam-Score: -3.058 X-Spam-Level: X-Spam-Status: No, score=-3.058 tagged_above=-999 required=5 tests=[AWL=0.540, BAYES_00=-2.599, HTML_MESSAGE=0.001, RCVD_IN_DNSWL_LOW=-1] Received: from mail.ietf.org ([12.22.58.30]) by localhost (ietfa.amsl.com [127.0.0.1]) (amavisd-new, port 10024) with ESMTP id CRyH2jk1MbZS for ; Wed, 6 Nov 2013 04:17:28 -0800 (PST) Received: from smtpe1.intersmtp.com (smtp64.intersmtp.com [62.239.224.237]) by ietfa.amsl.com (Postfix) with ESMTP id 1E57A11E80EE for ; Wed, 6 Nov 2013 04:17:27 -0800 (PST) Received: from EVMHT67-UKRD.domain1.systemhost.net (10.36.3.104) by RDW083A008ED64.smtp-e4.hygiene.service (10.187.98.13) with Microsoft SMTP Server (TLS) id 8.3.298.1; Wed, 6 Nov 2013 12:17:26 +0000 Received: from EMV64-UKRD.domain1.systemhost.net ([169.254.1.106]) by EVMHT67-UKRD.domain1.systemhost.net ([10.36.3.104]) with mapi; Wed, 6 Nov 2013 12:17:25 +0000 From: To: , Date: Wed, 6 Nov 2013 12:17:24 +0000 Thread-Topic: Comments on draft-burrbridge-lmap-information-model-01 Thread-Index: Ac7a6HZ5T6s7aqiQTHydIbHLJBrc7AAATW/g Message-ID: References: <9966516C6EB5FC4381E05BF80AA55F7720B35C@US70UWXCHMBA05.zam.alcatel-lucent.com> In-Reply-To: <9966516C6EB5FC4381E05BF80AA55F7720B35C@US70UWXCHMBA05.zam.alcatel-lucent.com> Accept-Language: en-US, en-GB Content-Language: en-US X-MS-Has-Attach: X-MS-TNEF-Correlator: acceptlanguage: en-US, en-GB Content-Type: multipart/alternative; boundary="_000_ED51D9282D1D3942B9438CA8F3372EB72C310E7BFFEMV64UKRDdoma_" MIME-Version: 1.0 Subject: Re: [lmap] Comments on draft-burrbridge-lmap-information-model-01 X-BeenThere: lmap@ietf.org X-Mailman-Version: 2.1.12 Precedence: list List-Id: Large Scale Measurement of Access network Performance List-Unsubscribe: , List-Archive: List-Post: List-Help: List-Subscribe: , X-List-Received-Date: Wed, 06 Nov 2013 12:17:33 -0000 --_000_ED51D9282D1D3942B9438CA8F3372EB72C310E7BFFEMV64UKRDdoma_ Content-Type: text/plain; charset="us-ascii" Content-Transfer-Encoding: quoted-printable Thanks, we'll pick up those points in the next edit. Regarding the test interface, what sort of "more opaque options" would you = like to see? If you just mean other undefined options, then yes, the option= s can be extended by the user beyond any standardised parameters, largely d= epending on the test being configured. Trevor. From: lmap-bounces@ietf.org [mailto:lmap-bounces@ietf.org] On Behalf Of Car= ey, Timothy (Timothy) Sent: 06 November 2013 12:05 To: lmap@ietf.org Subject: [lmap] Comments on draft-burrbridge-lmap-information-model-01 Trevor, I was utilizing this draft for a submission into BBF Q4 meeting and had a c= ouple of comments and a question. 1) In section 2.4 The Report Channel Names description is the same as = the Measurement Task Configuration Names; probably a copy paste error 2) In section 2.2 The pre-configuration information is still types as = a MAC address; in the BBF we use a device identifier; could I suggest we ch= ange this to Device Identifier and a URN. 3) In section 2.4 the Measurement Task options now has a sub-attribute= of Interface Name - which is OK, but I wanted to know if this was the only= option allowed or their will be more opaque options along with this explic= it option. BTW - Just a comment; I did notice that the major change in the draft was a= n introduction of the channel concept; it does seem alittle overkill to hav= e separate Instruction, Logging and Status channels since the same MA is ta= lking to the same Controller. I guess we are weighing flexibility with comp= lexity; remember these MAs could be quite constrained... BR, Tim --_000_ED51D9282D1D3942B9438CA8F3372EB72C310E7BFFEMV64UKRDdoma_ Content-Type: text/html; charset="us-ascii" Content-Transfer-Encoding: quoted-printable

Thanks, we’ll pick up those points in the next edit.

=  

R= egarding the test interface, what sort of “more opaque options”= would you like to see? If you just mean other undefined options, then yes,= the options can be extended by the user beyond any standardised parameters= , largely depending on the test being configured.

 

=

Trevor.

 

From:= lmap-bounces@ietf.org [mailto:lmap-bounces@ietf.org] On Behalf Of = Carey, Timothy (Timothy)
Sent: 06 November 2013 12:05
T= o: lmap@ietf.org
Subject: [lmap] Comments on draft-burrbridge= -lmap-information-model-01

 

Trevor,<= o:p>

 

I was utilizing this = draft for a submission into BBF Q4 meeting and had a couple of comments and= a question.

<= o:p> 

<= span style=3D'mso-list:Ignore'>1) &= nbsp;    I= n section 2.2 The pre-configuration information is still types as a MAC add= ress; in the BBF we use a device identifier; could I suggest we change this= to Device Identifier and a URN.

3)      = In section 2.4 the Measurement Task opt= ions now has a sub-attribute of Interface Name – which is OK, but I w= anted to know if this was the only option allowed or their will be more opa= que options along with this explicit option.

 

 

<= span lang=3DEN-US>BTW – Just a comment; I did notice that the major c= hange in the draft was an introduction of the channel concept; it does seem= alittle overkill to have separate Instruction, Logging and Status channels= since the same MA is talking to the same Controller. I guess we are weighi= ng flexibility with complexity; remember these MAs could be quite constrain= ed…

 

 =

BR,

Tim

 

= --_000_ED51D9282D1D3942B9438CA8F3372EB72C310E7BFFEMV64UKRDdoma_-- From timothy.carey@alcatel-lucent.com Wed Nov 6 04:22:30 2013 Return-Path: X-Original-To: lmap@ietfa.amsl.com Delivered-To: lmap@ietfa.amsl.com Received: from localhost (localhost [127.0.0.1]) by ietfa.amsl.com (Postfix) with ESMTP id CE40221E80EA for ; Wed, 6 Nov 2013 04:22:30 -0800 (PST) X-Virus-Scanned: amavisd-new at amsl.com X-Spam-Flag: NO X-Spam-Score: -10.298 X-Spam-Level: X-Spam-Status: No, score=-10.298 tagged_above=-999 required=5 tests=[AWL=0.300, BAYES_00=-2.599, HTML_MESSAGE=0.001, RCVD_IN_DNSWL_HI=-8] Received: from mail.ietf.org ([12.22.58.30]) by localhost (ietfa.amsl.com [127.0.0.1]) (amavisd-new, port 10024) with ESMTP id xAsCXubC25wY for ; Wed, 6 Nov 2013 04:22:24 -0800 (PST) Received: from ihemail1.lucent.com (ihemail1.lucent.com [135.245.0.33]) by ietfa.amsl.com (Postfix) with ESMTP id CB70521E80FA for ; Wed, 6 Nov 2013 04:22:10 -0800 (PST) Received: from us70uusmtp3.zam.alcatel-lucent.com (h135-5-2-65.lucent.com [135.5.2.65]) by ihemail1.lucent.com (8.13.8/IER-o) with ESMTP id rA6CM0qC027891 (version=TLSv1/SSLv3 cipher=DHE-RSA-AES256-SHA bits=256 verify=FAIL); Wed, 6 Nov 2013 06:22:00 -0600 (CST) Received: from US70TWXCHHUB04.zam.alcatel-lucent.com (us70twxchhub04.zam.alcatel-lucent.com [135.5.2.36]) by us70uusmtp3.zam.alcatel-lucent.com (GMO) with ESMTP id rA6CLuWs012946 (version=TLSv1/SSLv3 cipher=AES128-SHA bits=128 verify=FAIL); Wed, 6 Nov 2013 07:21:56 -0500 Received: from US70UWXCHMBA05.zam.alcatel-lucent.com ([169.254.10.30]) by US70TWXCHHUB04.zam.alcatel-lucent.com ([135.5.2.36]) with mapi id 14.02.0247.003; Wed, 6 Nov 2013 07:21:56 -0500 From: "Carey, Timothy (Timothy)" To: "trevor.burbridge@bt.com" , "lmap@ietf.org" Thread-Topic: Comments on draft-burrbridge-lmap-information-model-01 Thread-Index: Ac7a6HZ5T6s7aqiQTHydIbHLJBrc7AAATW/gAAAja+A= Date: Wed, 6 Nov 2013 12:21:55 +0000 Message-ID: <9966516C6EB5FC4381E05BF80AA55F7720B43E@US70UWXCHMBA05.zam.alcatel-lucent.com> References: <9966516C6EB5FC4381E05BF80AA55F7720B35C@US70UWXCHMBA05.zam.alcatel-lucent.com> In-Reply-To: Accept-Language: en-US Content-Language: en-US X-MS-Has-Attach: X-MS-TNEF-Correlator: x-originating-ip: [135.5.27.18] Content-Type: multipart/alternative; boundary="_000_9966516C6EB5FC4381E05BF80AA55F7720B43EUS70UWXCHMBA05zam_" MIME-Version: 1.0 X-Scanned-By: MIMEDefang 2.57 on 135.245.2.33 Subject: Re: [lmap] Comments on draft-burrbridge-lmap-information-model-01 X-BeenThere: lmap@ietf.org X-Mailman-Version: 2.1.12 Precedence: list List-Id: Large Scale Measurement of Access network Performance List-Unsubscribe: , List-Archive: List-Post: List-Help: List-Subscribe: , X-List-Received-Date: Wed, 06 Nov 2013 12:22:30 -0000 --_000_9966516C6EB5FC4381E05BF80AA55F7720B43EUS70UWXCHMBA05zam_ Content-Type: text/plain; charset="us-ascii" Content-Transfer-Encoding: quoted-printable Trevor, Yes that is what I meant - so we can say that Interface name is a standardi= zed option. I will model it that way for BBF. BR, Tim From: trevor.burbridge@bt.com [mailto:trevor.burbridge@bt.com] Sent: Wednesday, November 06, 2013 6:17 AM To: Carey, Timothy (Timothy); lmap@ietf.org Subject: RE: Comments on draft-burrbridge-lmap-information-model-01 Thanks, we'll pick up those points in the next edit. Regarding the test interface, what sort of "more opaque options" would you = like to see? If you just mean other undefined options, then yes, the option= s can be extended by the user beyond any standardised parameters, largely d= epending on the test being configured. Trevor. From: lmap-bounces@ietf.org [mailto:lmap-bounces@ietf.org] On Behalf Of Car= ey, Timothy (Timothy) Sent: 06 November 2013 12:05 To: lmap@ietf.org Subject: [lmap] Comments on draft-burrbridge-lmap-information-model-01 Trevor, I was utilizing this draft for a submission into BBF Q4 meeting and had a c= ouple of comments and a question. 1) In section 2.4 The Report Channel Names description is the same as = the Measurement Task Configuration Names; probably a copy paste error 2) In section 2.2 The pre-configuration information is still types as = a MAC address; in the BBF we use a device identifier; could I suggest we ch= ange this to Device Identifier and a URN. 3) In section 2.4 the Measurement Task options now has a sub-attribute= of Interface Name - which is OK, but I wanted to know if this was the only= option allowed or their will be more opaque options along with this explic= it option. BTW - Just a comment; I did notice that the major change in the draft was a= n introduction of the channel concept; it does seem alittle overkill to hav= e separate Instruction, Logging and Status channels since the same MA is ta= lking to the same Controller. I guess we are weighing flexibility with comp= lexity; remember these MAs could be quite constrained... BR, Tim --_000_9966516C6EB5FC4381E05BF80AA55F7720B43EUS70UWXCHMBA05zam_ Content-Type: text/html; charset="us-ascii" Content-Transfer-Encoding: quoted-printable

Trevor,

 

Yes that is what I mea= nt – so we can say that Interface name is a standardized option. &nbs= p;I will model it that way for BBF.

 

BR,<= /p>

Tim<= /p>

 

From: trevor.b= urbridge@bt.com [mailto:trevor.burbridge@bt.com]
Sent: Wednesday, November 06, 2013 6:17 AM
To: Carey, Timothy (Timothy); lmap@ietf.org
Subject: RE: Comments on draft-burrbridge-lmap-information-model-01<= o:p>

 

Thanks,= we’ll pick up those points in the next edit.

&n= bsp;

Regardi= ng the test interface, what sort of “more opaque options” would= you like to see? If you just mean other undefined options, then yes, the o= ptions can be extended by the user beyond any standardised parameters, largely depending on the test being configured.

&n= bsp;

Trevor.=

&n= bsp;

From: lmap-bou= nces@ietf.org [mailto:lmap-bounces@ietf.org] On Behalf Of Carey, Timothy (Timothy)
Sent: 06 November 2013 12:05
To: lmap@ietf.org
Subject: [lmap] Comments on draft-burrbridge-lmap-information-model-= 01

 

Trevor,

 

I was utilizing this draft for a submission into BBF= Q4 meeting and had a couple of comments and a question.

 

1)      In section 2.4 The Report Channel Names description= is the same as the Measurement Task Configuration Names; probably a copy p= aste error

2)      In section 2.2 The pre-configuration information is= still types as a MAC address; in the BBF we use a device identifier; could= I suggest we change this to Device Identifier and a URN.

3)      In section 2.4 the Measurement Task options now has= a sub-attribute of Interface Name – which is OK, but I wanted to kno= w if this was the only option allowed or their will be more opaque options = along with this explicit option.

 

 

BTW – Just a comment; I did notice that the ma= jor change in the draft was an introduction of the channel concept; it does= seem alittle overkill to have separate Instruction, Logging and Status cha= nnels since the same MA is talking to the same Controller. I guess we are weighing flexibility with complexity; reme= mber these MAs could be quite constrained…

 

 

BR,

Tim

 

--_000_9966516C6EB5FC4381E05BF80AA55F7720B43EUS70UWXCHMBA05zam_-- From acmorton@att.com Wed Nov 6 05:36:25 2013 Return-Path: X-Original-To: lmap@ietfa.amsl.com Delivered-To: lmap@ietfa.amsl.com Received: from localhost (localhost [127.0.0.1]) by ietfa.amsl.com (Postfix) with ESMTP id 82A7311E81B6 for ; Wed, 6 Nov 2013 05:36:24 -0800 (PST) X-Virus-Scanned: amavisd-new at amsl.com X-Spam-Flag: NO X-Spam-Score: -106.131 X-Spam-Level: X-Spam-Status: No, score=-106.131 tagged_above=-999 required=5 tests=[AWL=0.468, BAYES_00=-2.599, RCVD_IN_DNSWL_MED=-4, USER_IN_WHITELIST=-100] Received: from mail.ietf.org ([12.22.58.30]) by localhost (ietfa.amsl.com [127.0.0.1]) (amavisd-new, port 10024) with ESMTP id PYk6dXb-S8VT for ; Wed, 6 Nov 2013 05:36:18 -0800 (PST) Received: from nbfkord-smmo07.seg.att.com (nbfkord-smmo07.seg.att.com [209.65.160.93]) by ietfa.amsl.com (Postfix) with ESMTP id 35C1111E81A9 for ; Wed, 6 Nov 2013 05:36:18 -0800 (PST) Received: from unknown [144.160.229.24] (EHLO nbfkord-smmo07.seg.att.com) by nbfkord-smmo07.seg.att.com(mxl_mta-6.15.0-1) with ESMTP id 2d54a725.2aaae022c940.998877.00-569.2792922.nbfkord-smmo07.seg.att.com (envelope-from ); Wed, 06 Nov 2013 13:36:18 +0000 (UTC) X-MXL-Hash: 527a45d21450da47-046da95763744266e10463a46dfc6e54715f3cc2 Received: from unknown [144.160.229.24] (EHLO alpi155.enaf.aldc.att.com) by nbfkord-smmo07.seg.att.com(mxl_mta-6.15.0-1) over TLS secured channel with ESMTP id ec54a725.0.998865.00-411.2792885.nbfkord-smmo07.seg.att.com (envelope-from ); Wed, 06 Nov 2013 13:36:15 +0000 (UTC) X-MXL-Hash: 527a45cf52c45e08-991f19b2f1a88c2327911867bbf48a783a376476 Received: from enaf.aldc.att.com (localhost [127.0.0.1]) by alpi155.enaf.aldc.att.com (8.14.5/8.14.5) with ESMTP id rA6DaDTx013569; Wed, 6 Nov 2013 08:36:14 -0500 Received: from mlpi408.sfdc.sbc.com (mlpi408.sfdc.sbc.com [130.9.128.240]) by alpi155.enaf.aldc.att.com (8.14.5/8.14.5) with ESMTP id rA6Da3ur013417 (version=TLSv1/SSLv3 cipher=DHE-RSA-AES256-SHA bits=256 verify=NO); Wed, 6 Nov 2013 08:36:06 -0500 Received: from clpi183.sldc.sbc.com (clpi183.sldc.sbc.com [135.41.1.46]) by mlpi408.sfdc.sbc.com (RSA Interceptor); Wed, 6 Nov 2013 13:35:54 GMT Received: from sldc.sbc.com (localhost [127.0.0.1]) by clpi183.sldc.sbc.com (8.14.5/8.14.5) with ESMTP id rA6DZrxi012516; Wed, 6 Nov 2013 07:35:53 -0600 Received: from mail-azure.research.att.com (mail-azure.research.att.com [135.207.255.18]) by clpi183.sldc.sbc.com (8.14.5/8.14.5) with ESMTP id rA6DZm2t012227; Wed, 6 Nov 2013 07:35:48 -0600 Received: from njfpsrvexg2.research.att.com (njfpsrvexg2.research.att.com [135.207.160.21]) by mail-azure.research.att.com (Postfix) with ESMTP id D775EE02EF; Wed, 6 Nov 2013 08:35:47 -0500 (EST) Received: from NJFPSRVEXG8.research.att.com ([fe80::cdea:b3f6:3efa:1841]) by njfpsrvexg2.research.att.com ([fe80::a158:97ea:81b0:43d9%14]) with mapi; Wed, 6 Nov 2013 08:35:48 -0500 From: "MORTON, ALFRED C (AL)" To: "philip.eardley@bt.com" , "frode.sorensen@npt.no" , "lmap@ietf.org" Date: Wed, 6 Nov 2013 08:35:35 -0500 Thread-Topic: LMAP regulator use case Thread-Index: Ac7VfioGb4cSexizQzSZrQa31mFanAFFb7bPABe4HpA= Message-ID: <2845723087023D4CB5114223779FA9C8ABF5C7D1@njfpsrvexg8.research.att.com> References: <793D91975B99224DA9777562FF192808A27468CB@exmbx01> In-Reply-To: Accept-Language: en-US Content-Language: en-US X-MS-Has-Attach: X-MS-TNEF-Correlator: acceptlanguage: en-US Content-Type: text/plain; charset="iso-8859-1" Content-Transfer-Encoding: quoted-printable MIME-Version: 1.0 X-RSA-Inspected: yes X-RSA-Classifications: public X-Spam: [F=0.2000000000; CM=0.500; S=0.200(2010122901)] X-MAIL-FROM: X-SOURCE-IP: [144.160.229.24] X-AnalysisOut: [v=2.0 cv=FZmAMuC6 c=1 sm=0 a=dhB6nF3YHL5t/Ixux6cINA==:17 a] X-AnalysisOut: [=ofMgfj31e3cA:10 a=BLceEmwcHowA:10 a=8nJEP1OIZ-IA:10 a=xqW] X-AnalysisOut: [C_Br6kY4A:10 a=zQP7CpKOAAAA:8 a=s3Kb7qWvp0UA:10 a=48vgC7mU] X-AnalysisOut: [AAAA:8 a=e9qsufxtAAAA:8 a=BIJ-Ur6GOMQ4ZAj2xc4A:9 a=wPNLvfG] X-AnalysisOut: [TeEIA:10 a=lZB815dzVvQA:10 a=W1qU_X6G3J8A:10 a=e8VhWz08hTM] X-AnalysisOut: [IR3xE:21 a=G85dsjlK9GRv8MwL:21] Subject: Re: [lmap] LMAP regulator use case X-BeenThere: lmap@ietf.org X-Mailman-Version: 2.1.12 Precedence: list List-Id: Large Scale Measurement of Access network Performance List-Unsubscribe: , List-Archive: List-Post: List-Help: List-Subscribe: , X-List-Received-Date: Wed, 06 Nov 2013 13:36:25 -0000 Hi Frode, I read your mail and particularly agree with Phil's comments excerpted belo= w: > 4.4 Detecting degradation of the Internet access > 4.5 Detecting degradation of specific applications > > I don't like these sections about testing for whether the ISP is being ne= t > neutral. > I think it would be very hard to design tests. > In part this is because net neutrality is more of a political concept tha= n > technical one (in my view) > > I tihnk some of the stuff in S4.4 is really part of 4.3, as it's a market > development issue (the internet has got slower, and doesnt meet the polic= y > objective of Superfast broadband by Year 2015 - or whatever)- If one form of the absence of neutrality is censorship, even censorship is hard to define and can include performance degradation according to Nick Feamster's presentation at the IRTF open meeting: http://www.ietf.org/proceedings/88/slides/slides-88-irtfopen-3.pptx I think it's fair to say this sort of detection testing is still a research= topic, Al > -----Original Message----- > From: lmap-bounces@ietf.org [mailto:lmap-bounces@ietf.org] On Behalf Of > philip.eardley@bt.com > Sent: Tuesday, November 05, 2013 9:42 PM > To: frode.sorensen@npt.no; lmap@ietf.org > Subject: Re: [lmap] LMAP regulator use case > > Frode > thanks very much for the comments - and it is excellent to have detailed > text suggestions. > > 2.2 > I think i'm basically OK with this replacing the current S2.2 (including > the subsections) > I suggest that it should mention comparability in the final para (as well > as in the penultimate one) - personally I think this is the most importan= t > characteristic from a regulator's point of view, so that it's fair to > compare measurements they make across the different ISPs. > the one thing in the current S2.2 (actually in S2.2.3) that I think is > worth keeping is a mention of fixed and mobile. I think this could be > handled by adding to the Intro (S1) something like: "Measurements will > take place on both fixed and mobile access" > > > Happy to add a new chapter 4, but comments about some of your sub use > cases > > 4.1 Promoting competition through transparency > >>For competition to successfully discipline operators' behaviour in the > interests of their customers, end users must be fully aware of the > characteristics of the ISPs' access offers. > "must be fully aware" over-states things. > > >>The system can be operated by a regulator or a measurement provider. > add "on behalf of the regulator" > > >> However, the quality of an Internet access service is also determined > by the connectivity to the rest of the Internet. Therefore test > configurations which measure beyond the ISP's own network may also be > relevant in some cases. > true. perhaps worth mentioning that the ISPs can influence this to some > extent - how well they're connected to the Internet, how they cache > popular content etc. > > 4.2 Promoting end user empowerment > i think this can be deleted. this is the end user use case, not the > regulator use case. > > 4.3 Monitoring overall market development > good for me > > 4.4 Detecting degradation of the Internet access > 4.5 Detecting degradation of specific applications > > I don't like these sections about testing for whether the ISP is being ne= t > neutral. > I think it would be very hard to design tests. > In part this is because net neutrality is more of a political concept tha= n > technical one (in my view) > > i tihnk some of the stuff in S4.4 is really part of 4.3, as it's a market > development issue (the internet has got slower, and doesnt meet the polic= y > objective of Superfast broadband by Year 2015 - or whatever)- > > best wishes > phil > > ________________________________________ > From: lmap-bounces@ietf.org [lmap-bounces@ietf.org] On Behalf Of S=F8rens= en, > Frode [frode.sorensen@npt.no] > Sent: 30 October 2013 14:55 > To: lmap@ietf.org > Subject: [lmap] LMAP regulator use case > > Marc, Phil, Trevor, all, > > I have prepared a proposed description of a regulator use case for the > LMAP use case ID. I noticed the "advertisement" for further input from > regulators for the use case a couple of weeks ago. Finally I have managed > to find sufficient time to contribute to the ID, supported by a couple of > "regulator friends" from the Net Neutrality Working Group of the BEREC > organization, representing European national regulators. The proposal > consists of two parts; a modification of section 2.2 "Regulators", plus a > new chapter after chapter 3 "Details of ISP use case", titled "Details of > regulator use case". > > Thanks, > Frode > > > Proposed update: > > 2.2 Regulators > ------------------- > Regulators in jurisdictions around the world are responding to consumers' > adoption of Internet access services for traditional telecommunications > and media services by promoting competition among providers of electronic > communications, to ensure that users derive maximum benefit in terms of > choice, price, and quality. > > Some jurisdictions have responded to a need for greater information about > Internet access service performance in the development of regulatory > policies and approaches for broadband technologies by developing large- > scale measurement programs. Programs such as the U.S. Federal > Communications Commission's Measuring Broadband America, European > Commission's Quality of Broadband Services in the EU reports and a growin= g > list of other programs employ a diverse set of operational and technical > approaches to gathering data to perform analysis and reporting on diverse > aspects of broadband performance. > > While each jurisdiction responds to distinct consumer, industry, and > regulatory concerns, much commonality exists in the need to produce > datasets that are able to compare multiple Internet access service > providers, diverse technical solutions, geographic and regional > distributions, and marketed and provisioned levels and combinations of > broadband Internet access services. In some jurisdictions, the role of > measuring is provided by a measurement provider. > > Measurement providers measure network performance from users towards > multiple content and application providers, included dedicated test > measurement servers, to show a performance of the actual Internet access > service provided by different ISPs. Users need to know the performance > that are achieving from their own ISP. In addition, they need to know the > performance of other ISPs of same location as background information for > selecting their ISP. Measurement providers will provide measurement > results with associated measurement methods and measurement metrics. > > From a consumer perspective, the differentiation between fixed and mobile > (cellular) Internet access services is blurring as the applications used > are very similar. Hence, similar measurements will take place on both > fixed and mobile Internet access services. > > Regulators role in the development and enforcement of broadband Internet > access service policies also require that the measurement approaches meet > a high level of verifiability, accuracy and provider-independence to > support valid and meaningful comparisons of Internet access service > performance > > LMAP standards could answer regulators shared needs by providing scalable= , > cost-effective, scientifically robust solutions to the measurement and > collection of broadband Internet access service performance information. > > > Proposed new chapter: > > 4 Details of regulator use case > --------------------------------------- > 4.1 Promoting competition through transparency > --------------------------------------------------------------- > Competition plays a vital role in regulation of the electronic > communications markets. For competition to successfully discipline > operators' behaviour in the interests of their customers, end users must > be fully aware of the characteristics of the ISPs' access offers. In some > jurisdictions regulators mandate transparent information made available > about service offers. > > End users need effective transparency to be able to make informed choices > throughout the different stages of their relationship with ISPs, when > selecting Internet access service offers, and when considering switching > service offer within an ISP or to an alternative ISP. Quality information > about service offers can be provided based on performance measurements of > the services provided in the market. > > A set of measurement parameters and associated measurement methods are > used over time, e.g. speed, delay, and jitter. Then the measurement raw > data are collected and go through statistical post-processing before the > results can be published in an Internet access service quality index to > facilitate end users' choice of service provider and offer. > > In order to make information on offerings more meaningful and comparable > to end users, common frames of reference on Internet access service > quality metrics are essential. An example would be that in relation to > access speed, the actual download and upload speeds are measured and > published, not only the maximum speed. > > A measurement system that monitor Internet access services and collect > quality information can typically consist of a number of measurement > probes and one or more test servers located at peering points. The system > can be operated by a regulator or a measurement provider. Number and > distribution of probes follows specific requirements depending on the > scope and the desired statistical reliability of the measurement campaign= . > > In many test configurations, the measurement results will only reflect th= e > performance of ISP's own network. However, the quality of an Internet > access service is also determined by the connectivity to the rest of the > Internet. Therefore test configurations which measure beyond the ISP's ow= n > network may also be relevant in some cases. > > > 4.2 Promoting end user empowerment > -------------------------------------------------- > In addition to the advantages that end users receive from the competition > in the market, through availability of a plethora of ISPs and Internet > access service offers, facilitated by transparent quality information > about these offers at a general level, end users would further leverage o= n > information about specific services they subscribe to, or which they are > considering to subscribe to. > > Performance of Internet access service offers vary based on where the > service is provided (geographical and topological aspects), how it is > provided (e.g. access technology and ISP's interconnection to the > Internet) and when it is provided (variation through the day and over the > week). Therefore there is a need to obtain more detailed quality > information about specific service offers. > > Quality measurement tools can be made available for end users to monitor > their Internet access service, giving information about the speed and > other quality metrics of the subscribed service. Speed meters and similar > tools can be provided by regulators or other third parties, but also > directly by the ISP for the benefit of the subscriber. > > Such measurement tools assist end users in verifying the performance of > the Internet access service provided, comparing the measurement results > achieved with the information given in the contract. If the contracted > service quality level is not met, end users can use these results when > complaining to the ISP to achieve improvement and compensation. > > It may also help end users to assess whether they have chosen the package > which best suits their specific needs. With their performance results at > hand, they can benefit from quality evaluation guides, which offer > comparisons based on their specific usage profiles. This may enable them > to check whether their current subscription is the most appropriate, and > assist them to switch package or provider if it is in their best interest= . > > When provided by regulators and other third parties, such quality > measurement tools can also be used to compare performance between ISPs. I= t > should, though, be recognised that measurement results can be affected by > factors outside the ISP's control, related for instance with local networ= k > connection (e.g. Wi-Fi), equipment and software being used. > > > 4.3 Monitoring overall market development > --------------------------------------------------------- > Governments sometimes set strategic goals for high-speed Internet access > penetration as an important component of the economic, cultural and socia= l > development of the society. To evaluate the effect of the stimulated > growth over time, broadband Internet access take-up and penetration of > high-speed access can be monitored through measurement campaigns. > > An example of such an initiative is the "Digital Agenda for Europe" which > was adopted in 2010, to achieve universal broadband access. The goal is t= o > achieve by 2020, access for all Europeans to Internet access speeds of 30 > Mbps or above, and 50% or more of European households subscribing to > Internet connections above 100 Mbps. > > To monitor actual broadband Internet access performance in a specific > country or a region, extensive measurement campaigns are needed. A panel > can be built based on operators and packages in the market, spread over > urban, suburban and rural areas. Probes can then be distributed to the > participants of the campaign. > > Periodic tests running on the probes can for example measure actual speed > at peak and off-peak hours, but also other detailed quality metrics like > delay and jitter. Collected data goes afterwards through statistical > analysis, deriving estimates for the whole population which can then be > presented and published regularly. > > Using a harmonized or standardised measurement methodology, or even a > common quality measurement platform, measurement results could also be > used for benchmarking of providers and/or countries. > > > 4.4 Detecting degradation of the Internet access > -------------------------------------------------------------- > Regulation related to net neutrality and the open Internet has been > introduced in some jurisdictions' Internet policy, and monitoring methods > for detection of potential net neutrality breaches is needed to follow up > such regulations with concrete actions. > > > Net neutrality is typically related to equal treatment of traffic > transmitted over the Internet access service, while other traffic > transmitted in parallel over the end user's broadband connection e.g. to > provide IPTV in IP networks separated from the Internet (also referred to > as "closed IP networks") can be exempted from net neutrality regulation. > Services provided in parallel to the Internet access services are referre= d > to as specialised services (ref. BEREC and FCC). > > In order for such regulatory service architecture to protect net > neutrality and the open Internet, it is essential that the specialised > services are not provided at the expense of the Internet access service. > Therefore regulators may need to measure performance of the Internet > access service over time, and thereby detect whether this service is > becoming degraded. > > When regulators monitor this development to ensure net neutrality and the > open Internet, they can check e.g. whether Internet access service > performance and level of quality reflect advances in technology and that > this is not impaired by specialised services. Comparison between ISPs or > between different countries may also be relevant for this kind of > evaluation. > > Based on such Internet access service quality monitoring, regulators will > typically report results and findings regularly, e.g. on an annual basis. > In some jurisdictions regulators may also eventually impose minimum > quality of service requirements on ISPs in order to prevent degradation o= f > Internet access services. > > > 4.5 Detecting degradation of specific applications > --------------------------------------------------------------- > In the regulatory context related to net neutrality and the open Internet= , > in addition to the monitoring to detect potential degradation of the > access service as a whole described above, it is essential to perform > monitoring to detect potential degradation of individual applications > using the Internet access. > > Since net neutrality relates to equal treatment of traffic transmitted > over the Internet access service, the treatment of different applications= ' > traffic is in the core of net neutrality. When traffic originating from > different applications are treated independent of which application each > IP packet belongs to, this is referred to as application agnosticism. > > Examples of departure from application agnosticism are blocking or > throttling of traffic from specific applications, but also preferential > treatment of specific applications. If traffic is forwarded at different > priority levels, traffic having higher priority level implies that other > traffic has lower priority. Detection of such traffic management practice= s > does not necessarily imply breaches to net neutrality, but these > observations can be used as an input to the regulatory evaluation of the > practice. > > When regulators monitor traffic management practices to ensure net > neutrality and the open Internet, they can check compliance with criteria > defining what is considered reasonable traffic management. Measurement > systems designed to monitor this kind of effects need to send application= - > specific traffic and then measure in detail the behaviour of the differen= t > packets receive when transferred over the Internet access service. > > Today there exist measurement tools that can detect differentiated > treatment of individual applications. Another method that can be used is > to measure performance at application layer, e.g. assisted by content and > application providers, but also passive measurements are foreseen as a > promising approach for monitoring of application-specific treatment. > > _______________________________________________ > lmap mailing list > lmap@ietf.org > https://www.ietf.org/mailman/listinfo/lmap > _______________________________________________ > lmap mailing list > lmap@ietf.org > https://www.ietf.org/mailman/listinfo/lmap From jamesmilleresquire@gmail.com Wed Nov 6 06:42:49 2013 Return-Path: X-Original-To: lmap@ietfa.amsl.com Delivered-To: lmap@ietfa.amsl.com Received: from localhost (localhost [127.0.0.1]) by ietfa.amsl.com (Postfix) with ESMTP id 01D7321E8121 for ; Wed, 6 Nov 2013 06:42:49 -0800 (PST) X-Virus-Scanned: amavisd-new at amsl.com X-Spam-Flag: NO X-Spam-Score: -2.599 X-Spam-Level: X-Spam-Status: No, score=-2.599 tagged_above=-999 required=5 tests=[BAYES_00=-2.599, HTML_MESSAGE=0.001, NO_RELAYS=-0.001] Received: from mail.ietf.org ([12.22.58.30]) by localhost (ietfa.amsl.com [127.0.0.1]) (amavisd-new, port 10024) with ESMTP id OweGm61Dc2yl for ; Wed, 6 Nov 2013 06:42:46 -0800 (PST) Received: from mail-wi0-x233.google.com (mail-wi0-x233.google.com [IPv6:2a00:1450:400c:c05::233]) by ietfa.amsl.com (Postfix) with ESMTP id 2BDB811E81A2 for ; Wed, 6 Nov 2013 06:42:43 -0800 (PST) Received: by mail-wi0-f179.google.com with SMTP id hm4so3797897wib.12 for ; Wed, 06 Nov 2013 06:42:34 -0800 (PST) DKIM-Signature: v=1; a=rsa-sha256; c=relaxed/relaxed; d=gmail.com; s=20120113; h=mime-version:in-reply-to:references:date:message-id:subject:from:to :cc:content-type; bh=Qy3RIm/czjvi8MOwbNkUp0/jr61BOTo2ScB1KkfkT/A=; b=0VPZ4t/ySXnbImmEt7qzGtycuaLcoNpNtnU5wriLTtmptSQxecXzOHmX3Ogh/x+mLb 8O+Fij1d1S00yQTzRM22DFVaZzGoYrzB1OxjGuFikqOLkF5zAB/Y52HAEM58O/mo6ATp HFSZ6XWaQz3TqXYk5KazequD82H5Cw2UshScsZKl4Ch37GCFYn7IPA3qQAU7xtva8O95 CqKIFpCGPnlIfZVXqiI/703nZ+j3dYU44pLi+XqqSCrbYmo2ok16syT33qq6UC3oJfW7 ctRShfq72ELZjV2D/mPWDWm0QnSXupoMinMmfy4hl8VuQQvIbarkneJj7CYawkT7gJUw rcKA== MIME-Version: 1.0 X-Received: by 10.180.12.14 with SMTP id u14mr2701181wib.63.1383748954373; Wed, 06 Nov 2013 06:42:34 -0800 (PST) Received: by 10.180.211.7 with HTTP; Wed, 6 Nov 2013 06:42:34 -0800 (PST) Received: by 10.180.211.7 with HTTP; Wed, 6 Nov 2013 06:42:34 -0800 (PST) In-Reply-To: <2845723087023D4CB5114223779FA9C8ABF5C7D1@njfpsrvexg8.research.att.com> References: <793D91975B99224DA9777562FF192808A27468CB@exmbx01> <2845723087023D4CB5114223779FA9C8ABF5C7D1@njfpsrvexg8.research.att.com> Date: Wed, 6 Nov 2013 09:42:34 -0500 Message-ID: From: James Miller To: "MORTON JR. ALFRED, (AL)" Content-Type: multipart/alternative; boundary=001a11c23ed2e2453b04ea8328c6 Cc: philip.eardley@bt.com, "lmap@ietf.org" , frode.sorensen@npt.no Subject: Re: [lmap] LMAP regulator use case X-BeenThere: lmap@ietf.org X-Mailman-Version: 2.1.12 Precedence: list List-Id: Large Scale Measurement of Access network Performance List-Unsubscribe: , List-Archive: List-Post: List-Help: List-Subscribe: , X-List-Received-Date: Wed, 06 Nov 2013 14:42:49 -0000 --001a11c23ed2e2453b04ea8328c6 Content-Type: text/plain; charset=ISO-8859-1 Content-Transfer-Encoding: quoted-printable I agree with Frode's comments on the 2.2 points. I would add the quality of the comparability of data is crucial for a regulators use and extends to technical neutrality across the access link technologies as well, eg ability to compare docsis, DSL, fiber, fixed wireless or other solutions. Most studies I am familiar with include present and post processing techniques to support comparisons across major broadband technologies that include out of scope topics. With 4.4 or 4.5 I agree there are a mix of views but would suggest there are production quality tools as well as more research level quality topics. I believe many are likely to extend to discussion of ippm metrics. In addition to the tools mentioned, nick weaver's netalizer suite is an example that tests for easy things (like port blocking) and harder more subtle effects (like throttling, caching and proxy effects), and would also mention work by bitag and other groups trying to reach technical consensus on description of many traffic management practices. It is an evolving space but something relevant for regulators use case description. On Nov 6, 2013 8:36 AM, "MORTON, ALFRED C (AL)" wrote: > Hi Frode, > > I read your mail and particularly agree with Phil's comments excerpted > below: > > > 4.4 Detecting degradation of the Internet access > > 4.5 Detecting degradation of specific applications > > > > I don't like these sections about testing for whether the ISP is being > net > > neutral. > > I think it would be very hard to design tests. > > In part this is because net neutrality is more of a political concept > than > > technical one (in my view) > > > > I tihnk some of the stuff in S4.4 is really part of 4.3, as it's a mark= et > > development issue (the internet has got slower, and doesnt meet the > policy > > objective of Superfast broadband by Year 2015 - or whatever)- > > If one form of the absence of neutrality is censorship, even > censorship is hard to define and can include performance degradation > according to Nick Feamster's presentation at the IRTF open meeting: > http://www.ietf.org/proceedings/88/slides/slides-88-irtfopen-3.pptx > > I think it's fair to say this sort of detection testing is still a > research topic, > Al > > > -----Original Message----- > > From: lmap-bounces@ietf.org [mailto:lmap-bounces@ietf.org] On Behalf Of > > philip.eardley@bt.com > > Sent: Tuesday, November 05, 2013 9:42 PM > > To: frode.sorensen@npt.no; lmap@ietf.org > > Subject: Re: [lmap] LMAP regulator use case > > > > Frode > > thanks very much for the comments - and it is excellent to have detaile= d > > text suggestions. > > > > 2.2 > > I think i'm basically OK with this replacing the current S2.2 (includin= g > > the subsections) > > I suggest that it should mention comparability in the final para (as we= ll > > as in the penultimate one) - personally I think this is the most > important > > characteristic from a regulator's point of view, so that it's fair to > > compare measurements they make across the different ISPs. > > the one thing in the current S2.2 (actually in S2.2.3) that I think is > > worth keeping is a mention of fixed and mobile. I think this could be > > handled by adding to the Intro (S1) something like: "Measurements will > > take place on both fixed and mobile access" > > > > > > Happy to add a new chapter 4, but comments about some of your sub use > > cases > > > > 4.1 Promoting competition through transparency > > >>For competition to successfully discipline operators' behaviour in th= e > > interests of their customers, end users must be fully aware of the > > characteristics of the ISPs' access offers. > > "must be fully aware" over-states things. > > > > >>The system can be operated by a regulator or a measurement provider. > > add "on behalf of the regulator" > > > > >> However, the quality of an Internet access service is also determin= ed > > by the connectivity to the rest of the Internet. Therefore test > > configurations which measure beyond the ISP's own network may also be > > relevant in some cases. > > true. perhaps worth mentioning that the ISPs can influence this to some > > extent - how well they're connected to the Internet, how they cache > > popular content etc. > > > > 4.2 Promoting end user empowerment > > i think this can be deleted. this is the end user use case, not the > > regulator use case. > > > > 4.3 Monitoring overall market development > > good for me > > > > 4.4 Detecting degradation of the Internet access > > 4.5 Detecting degradation of specific applications > > > > I don't like these sections about testing for whether the ISP is being > net > > neutral. > > I think it would be very hard to design tests. > > In part this is because net neutrality is more of a political concept > than > > technical one (in my view) > > > > i tihnk some of the stuff in S4.4 is really part of 4.3, as it's a mark= et > > development issue (the internet has got slower, and doesnt meet the > policy > > objective of Superfast broadband by Year 2015 - or whatever)- > > > > best wishes > > phil > > > > ________________________________________ > > From: lmap-bounces@ietf.org [lmap-bounces@ietf.org] On Behalf Of > S=F8rensen, > > Frode [frode.sorensen@npt.no] > > Sent: 30 October 2013 14:55 > > To: lmap@ietf.org > > Subject: [lmap] LMAP regulator use case > > > > Marc, Phil, Trevor, all, > > > > I have prepared a proposed description of a regulator use case for the > > LMAP use case ID. I noticed the "advertisement" for further input from > > regulators for the use case a couple of weeks ago. Finally I have manag= ed > > to find sufficient time to contribute to the ID, supported by a couple = of > > "regulator friends" from the Net Neutrality Working Group of the BEREC > > organization, representing European national regulators. The proposal > > consists of two parts; a modification of section 2.2 "Regulators", plus= a > > new chapter after chapter 3 "Details of ISP use case", titled "Details = of > > regulator use case". > > > > Thanks, > > Frode > > > > > > Proposed update: > > > > 2.2 Regulators > > ------------------- > > Regulators in jurisdictions around the world are responding to consumer= s' > > adoption of Internet access services for traditional telecommunications > > and media services by promoting competition among providers of electron= ic > > communications, to ensure that users derive maximum benefit in terms of > > choice, price, and quality. > > > > Some jurisdictions have responded to a need for greater information abo= ut > > Internet access service performance in the development of regulatory > > policies and approaches for broadband technologies by developing large- > > scale measurement programs. Programs such as the U.S. Federal > > Communications Commission's Measuring Broadband America, European > > Commission's Quality of Broadband Services in the EU reports and a > growing > > list of other programs employ a diverse set of operational and technica= l > > approaches to gathering data to perform analysis and reporting on diver= se > > aspects of broadband performance. > > > > While each jurisdiction responds to distinct consumer, industry, and > > regulatory concerns, much commonality exists in the need to produce > > datasets that are able to compare multiple Internet access service > > providers, diverse technical solutions, geographic and regional > > distributions, and marketed and provisioned levels and combinations of > > broadband Internet access services. In some jurisdictions, the role of > > measuring is provided by a measurement provider. > > > > Measurement providers measure network performance from users towards > > multiple content and application providers, included dedicated test > > measurement servers, to show a performance of the actual Internet acces= s > > service provided by different ISPs. Users need to know the performance > > that are achieving from their own ISP. In addition, they need to know t= he > > performance of other ISPs of same location as background information fo= r > > selecting their ISP. Measurement providers will provide measurement > > results with associated measurement methods and measurement metrics. > > > > From a consumer perspective, the differentiation between fixed and mobi= le > > (cellular) Internet access services is blurring as the applications use= d > > are very similar. Hence, similar measurements will take place on both > > fixed and mobile Internet access services. > > > > Regulators role in the development and enforcement of broadband Interne= t > > access service policies also require that the measurement approaches me= et > > a high level of verifiability, accuracy and provider-independence to > > support valid and meaningful comparisons of Internet access service > > performance > > > > LMAP standards could answer regulators shared needs by providing > scalable, > > cost-effective, scientifically robust solutions to the measurement and > > collection of broadband Internet access service performance information= . > > > > > > Proposed new chapter: > > > > 4 Details of regulator use case > > --------------------------------------- > > 4.1 Promoting competition through transparency > > --------------------------------------------------------------- > > Competition plays a vital role in regulation of the electronic > > communications markets. For competition to successfully discipline > > operators' behaviour in the interests of their customers, end users mus= t > > be fully aware of the characteristics of the ISPs' access offers. In so= me > > jurisdictions regulators mandate transparent information made available > > about service offers. > > > > End users need effective transparency to be able to make informed choic= es > > throughout the different stages of their relationship with ISPs, when > > selecting Internet access service offers, and when considering switchin= g > > service offer within an ISP or to an alternative ISP. Quality informati= on > > about service offers can be provided based on performance measurements = of > > the services provided in the market. > > > > A set of measurement parameters and associated measurement methods are > > used over time, e.g. speed, delay, and jitter. Then the measurement raw > > data are collected and go through statistical post-processing before th= e > > results can be published in an Internet access service quality index to > > facilitate end users' choice of service provider and offer. > > > > In order to make information on offerings more meaningful and comparabl= e > > to end users, common frames of reference on Internet access service > > quality metrics are essential. An example would be that in relation to > > access speed, the actual download and upload speeds are measured and > > published, not only the maximum speed. > > > > A measurement system that monitor Internet access services and collect > > quality information can typically consist of a number of measurement > > probes and one or more test servers located at peering points. The syst= em > > can be operated by a regulator or a measurement provider. Number and > > distribution of probes follows specific requirements depending on the > > scope and the desired statistical reliability of the measurement > campaign. > > > > In many test configurations, the measurement results will only reflect > the > > performance of ISP's own network. However, the quality of an Internet > > access service is also determined by the connectivity to the rest of th= e > > Internet. Therefore test configurations which measure beyond the ISP's > own > > network may also be relevant in some cases. > > > > > > 4.2 Promoting end user empowerment > > -------------------------------------------------- > > In addition to the advantages that end users receive from the competiti= on > > in the market, through availability of a plethora of ISPs and Internet > > access service offers, facilitated by transparent quality information > > about these offers at a general level, end users would further leverage > on > > information about specific services they subscribe to, or which they ar= e > > considering to subscribe to. > > > > Performance of Internet access service offers vary based on where the > > service is provided (geographical and topological aspects), how it is > > provided (e.g. access technology and ISP's interconnection to the > > Internet) and when it is provided (variation through the day and over t= he > > week). Therefore there is a need to obtain more detailed quality > > information about specific service offers. > > > > Quality measurement tools can be made available for end users to monito= r > > their Internet access service, giving information about the speed and > > other quality metrics of the subscribed service. Speed meters and simil= ar > > tools can be provided by regulators or other third parties, but also > > directly by the ISP for the benefit of the subscriber. > > > > Such measurement tools assist end users in verifying the performance of > > the Internet access service provided, comparing the measurement results > > achieved with the information given in the contract. If the contracted > > service quality level is not met, end users can use these results when > > complaining to the ISP to achieve improvement and compensation. > > > > It may also help end users to assess whether they have chosen the packa= ge > > which best suits their specific needs. With their performance results a= t > > hand, they can benefit from quality evaluation guides, which offer > > comparisons based on their specific usage profiles. This may enable the= m > > to check whether their current subscription is the most appropriate, an= d > > assist them to switch package or provider if it is in their best > interest. > > > > When provided by regulators and other third parties, such quality > > measurement tools can also be used to compare performance between ISPs. > It > > should, though, be recognised that measurement results can be affected = by > > factors outside the ISP's control, related for instance with local > network > > connection (e.g. Wi-Fi), equipment and software being used. > > > > > > 4.3 Monitoring overall market development > > --------------------------------------------------------- > > Governments sometimes set strategic goals for high-speed Internet acces= s > > penetration as an important component of the economic, cultural and > social > > development of the society. To evaluate the effect of the stimulated > > growth over time, broadband Internet access take-up and penetration of > > high-speed access can be monitored through measurement campaigns. > > > > An example of such an initiative is the "Digital Agenda for Europe" whi= ch > > was adopted in 2010, to achieve universal broadband access. The goal is > to > > achieve by 2020, access for all Europeans to Internet access speeds of = 30 > > Mbps or above, and 50% or more of European households subscribing to > > Internet connections above 100 Mbps. > > > > To monitor actual broadband Internet access performance in a specific > > country or a region, extensive measurement campaigns are needed. A pane= l > > can be built based on operators and packages in the market, spread over > > urban, suburban and rural areas. Probes can then be distributed to the > > participants of the campaign. > > > > Periodic tests running on the probes can for example measure actual spe= ed > > at peak and off-peak hours, but also other detailed quality metrics lik= e > > delay and jitter. Collected data goes afterwards through statistical > > analysis, deriving estimates for the whole population which can then be > > presented and published regularly. > > > > Using a harmonized or standardised measurement methodology, or even a > > common quality measurement platform, measurement results could also be > > used for benchmarking of providers and/or countries. > > > > > > 4.4 Detecting degradation of the Internet access > > -------------------------------------------------------------- > > Regulation related to net neutrality and the open Internet has been > > introduced in some jurisdictions' Internet policy, and monitoring metho= ds > > for detection of potential net neutrality breaches is needed to follow = up > > such regulations with concrete actions. > > > > > > Net neutrality is typically related to equal treatment of traffic > > transmitted over the Internet access service, while other traffic > > transmitted in parallel over the end user's broadband connection e.g. t= o > > provide IPTV in IP networks separated from the Internet (also referred = to > > as "closed IP networks") can be exempted from net neutrality regulation= . > > Services provided in parallel to the Internet access services are > referred > > to as specialised services (ref. BEREC and FCC). > > > > In order for such regulatory service architecture to protect net > > neutrality and the open Internet, it is essential that the specialised > > services are not provided at the expense of the Internet access service= . > > Therefore regulators may need to measure performance of the Internet > > access service over time, and thereby detect whether this service is > > becoming degraded. > > > > When regulators monitor this development to ensure net neutrality and t= he > > open Internet, they can check e.g. whether Internet access service > > performance and level of quality reflect advances in technology and tha= t > > this is not impaired by specialised services. Comparison between ISPs o= r > > between different countries may also be relevant for this kind of > > evaluation. > > > > Based on such Internet access service quality monitoring, regulators wi= ll > > typically report results and findings regularly, e.g. on an annual basi= s. > > In some jurisdictions regulators may also eventually impose minimum > > quality of service requirements on ISPs in order to prevent degradation > of > > Internet access services. > > > > > > 4.5 Detecting degradation of specific applications > > --------------------------------------------------------------- > > In the regulatory context related to net neutrality and the open > Internet, > > in addition to the monitoring to detect potential degradation of the > > access service as a whole described above, it is essential to perform > > monitoring to detect potential degradation of individual applications > > using the Internet access. > > > > Since net neutrality relates to equal treatment of traffic transmitted > > over the Internet access service, the treatment of different > applications' > > traffic is in the core of net neutrality. When traffic originating from > > different applications are treated independent of which application eac= h > > IP packet belongs to, this is referred to as application agnosticism. > > > > Examples of departure from application agnosticism are blocking or > > throttling of traffic from specific applications, but also preferential > > treatment of specific applications. If traffic is forwarded at differen= t > > priority levels, traffic having higher priority level implies that othe= r > > traffic has lower priority. Detection of such traffic management > practices > > does not necessarily imply breaches to net neutrality, but these > > observations can be used as an input to the regulatory evaluation of th= e > > practice. > > > > When regulators monitor traffic management practices to ensure net > > neutrality and the open Internet, they can check compliance with criter= ia > > defining what is considered reasonable traffic management. Measurement > > systems designed to monitor this kind of effects need to send > application- > > specific traffic and then measure in detail the behaviour of the > different > > packets receive when transferred over the Internet access service. > > > > Today there exist measurement tools that can detect differentiated > > treatment of individual applications. Another method that can be used i= s > > to measure performance at application layer, e.g. assisted by content a= nd > > application providers, but also passive measurements are foreseen as a > > promising approach for monitoring of application-specific treatment. > > > > _______________________________________________ > > lmap mailing list > > lmap@ietf.org > > https://www.ietf.org/mailman/listinfo/lmap > > _______________________________________________ > > lmap mailing list > > lmap@ietf.org > > https://www.ietf.org/mailman/listinfo/lmap > _______________________________________________ > lmap mailing list > lmap@ietf.org > https://www.ietf.org/mailman/listinfo/lmap > --001a11c23ed2e2453b04ea8328c6 Content-Type: text/html; charset=ISO-8859-1 Content-Transfer-Encoding: quoted-printable

I agree with Frode's comments on the 2.2 points.=A0 I wo= uld add the quality of the comparability of data is crucial for a regulator= s use and extends to technical neutrality across the access link technologi= es as well, eg ability to compare docsis, DSL, fiber, fixed wireless or oth= er solutions.=A0 Most studies I am familiar with include present and post p= rocessing techniques to support comparisons across major broadband technolo= gies that include out of scope topics.

With 4.4 or 4.5 I agree there are a mix of views but would s= uggest there are production quality tools as well as more research level qu= ality topics.=A0 I believe many are likely to extend to discussion of ippm = metrics.=A0 In addition to the tools mentioned, nick weaver's netalizer= suite is an example that tests for easy things (like port blocking) and ha= rder more subtle effects (like throttling, caching and proxy effects), and = would also mention work by bitag and other groups trying to reach technical= consensus on description of many traffic management practices.=A0 It is an= evolving space but something relevant for regulators use case description.=

On Nov 6, 2013 8:36 AM, "MORTON, ALFRED C (= AL)" <acmorton@att.com> = wrote:
Hi Frode,

I read your mail and particularly agree with Phil's comments excerpted = below:

> 4.4 Detecting degradation of the Internet access
> 4.5 Detecting degradation of specific applications
>
> I don't like these sections about testing for whether the ISP is b= eing net
> neutral.
> I think it would be very hard to design tests.
> In part this is because net neutrality is more of a political concept = than
> technical one (in my view)
>
> I tihnk some of the stuff in S4.4 is really part of 4.3, as it's a= market
> development issue (the internet has got slower, and doesnt meet the po= licy
> objective of Superfast broadband by Year 2015 - or whatever)-

If one form of the absence of neutrality is censorship, even
censorship is hard to define and can include performance degradation
according to Nick Feamster's presentation at the IRTF open meeting:
http://www.ietf.org/proceedings/88/slides/slides-88-= irtfopen-3.pptx

I think it's fair to say this sort of detection testing is still a rese= arch topic,
Al

> -----Original Message-----
> From: lmap-bounces@ietf.org [mailto:lmap-bounces@ietf.org] On Behalf Of
> philip.eardley@bt.com
> Sent: Tuesday, November 05, 2013 9:42 PM
> To: frode.sorensen@npt.no= ; lmap@ietf.org
> Subject: Re: [lmap] LMAP regulator use case
>
> Frode
> thanks very much for the comments - and it is excellent to have detail= ed
> text suggestions.
>
> 2.2
> I think i'm basically OK with this replacing the current S2.2 (inc= luding
> the subsections)
> I suggest that it should mention comparability in the final para (as w= ell
> as in the penultimate one) - personally I think this is the most impor= tant
> characteristic from a regulator's point of view, so that it's = fair to
> compare measurements they make across the different ISPs.
> the one thing in the current S2.2 (actually in S2.2.3) that I think is=
> worth keeping is a mention of fixed and mobile. I think this could be<= br> > handled by adding to the Intro (S1) something like: "Measurements= will
> take place on both fixed and mobile access"
>
>
> Happy to add a new chapter 4, but comments about some of your sub use<= br> > cases
>
> 4.1 Promoting competition through transparency
> >>For competition to successfully discipline operators' beha= viour in the
> interests of their customers, end users must be fully aware of the
> characteristics of the ISPs' access offers.
> "must be fully aware" over-states things.
>
> >>The system can be operated by a regulator or a measurement pro= vider.
> add "on behalf of the regulator"
>
> >> =A0However, the quality of an Internet access service is also= determined
> by the connectivity to the rest of the Internet. Therefore test
> configurations which measure beyond the ISP's own network may also= be
> relevant in some cases.
> true. perhaps worth mentioning that the ISPs can influence this to som= e
> extent - how well they're connected to the Internet, how they cach= e
> popular content etc.
>
> 4.2 Promoting end user empowerment
> i think this can be deleted. this is the end user use case, not the > regulator use case.
>
> 4.3 Monitoring overall market development
> good for me
>
> 4.4 Detecting degradation of the Internet access
> 4.5 Detecting degradation of specific applications
>
> I don't like these sections about testing for whether the ISP is b= eing net
> neutral.
> I think it would be very hard to design tests.
> In part this is because net neutrality is more of a political concept = than
> technical one (in my view)
>
> i tihnk some of the stuff in S4.4 is really part of 4.3, as it's a= market
> development issue (the internet has got slower, and doesnt meet the po= licy
> objective of Superfast broadband by Year 2015 - or whatever)-
>
> best wishes
> phil
>
> ________________________________________
> From: lmap-bounces@ietf.org [lmap-bounces@ietf.org] On = Behalf Of S=F8rensen,
> Frode [frode.sorensen@npt.no<= /a>]
> Sent: 30 October 2013 14:55
> To: lmap@ietf.org
> Subject: [lmap] LMAP regulator use case
>
> Marc, Phil, Trevor, all,
>
> I have prepared a proposed description of a regulator use case for the=
> LMAP use case ID. I noticed the "advertisement" for further = input from
> regulators for the use case a couple of weeks ago. Finally I have mana= ged
> to find sufficient time to contribute to the ID, supported by a couple= of
> "regulator friends" from the Net Neutrality Working Group of= the BEREC
> organization, representing European national regulators. The proposal<= br> > consists of two parts; a modification of section 2.2 "Regulators&= quot;, plus a
> new chapter after chapter 3 "Details of ISP use case", title= d "Details of
> regulator use case".
>
> Thanks,
> Frode
>
>
> Proposed update:
>
> 2.2 Regulators
> -------------------
> Regulators in jurisdictions around the world are responding to consume= rs'
> adoption of Internet access services for traditional telecommunication= s
> and media services by promoting competition among providers of electro= nic
> communications, to ensure that users derive maximum benefit in terms o= f
> choice, price, and quality.
>
> Some jurisdictions have responded to a need for greater information ab= out
> Internet access service performance in the development of regulatory > policies and approaches for broadband technologies by developing large= -
> scale measurement programs. Programs such as the U.S. Federal
> Communications Commission's Measuring Broadband America, European<= br> > Commission's Quality of Broadband Services in the EU reports and a= growing
> list of other programs employ a diverse set of operational and technic= al
> approaches to gathering data to perform analysis and reporting on dive= rse
> aspects of broadband performance.
>
> While each jurisdiction responds to distinct consumer, industry, and > regulatory concerns, much commonality exists in the need to produce > datasets that are able to compare multiple Internet access service
> providers, diverse technical solutions, geographic and regional
> distributions, and marketed and provisioned levels and combinations of=
> broadband Internet access services. In some jurisdictions, the role of=
> measuring is provided by a measurement provider.
>
> Measurement providers measure network performance from users towards > multiple content and application providers, included dedicated test > measurement servers, to show a performance of the actual Internet acce= ss
> service provided by different ISPs. Users need to know the performance=
> that are achieving from their own ISP. In addition, they need to know = the
> performance of other ISPs of same location as background information f= or
> selecting their ISP. Measurement providers will provide measurement > results with associated measurement methods and measurement metrics. >
> From a consumer perspective, the differentiation between fixed and mob= ile
> (cellular) Internet access services is blurring as the applications us= ed
> are very similar. Hence, similar measurements will take place on both<= br> > fixed and mobile Internet access services.
>
> Regulators role in the development and enforcement of broadband Intern= et
> access service policies also require that the measurement approaches m= eet
> a high level of verifiability, accuracy and provider-independence to > support valid and meaningful comparisons of Internet access service > performance
>
> LMAP standards could answer regulators shared needs by providing scala= ble,
> cost-effective, scientifically robust solutions to the measurement and=
> collection of broadband Internet access service performance informatio= n.
>
>
> Proposed new chapter:
>
> 4 Details of regulator use case
> ---------------------------------------
> 4.1 Promoting competition through transparency
> ---------------------------------------------------------------
> Competition plays a vital role in regulation of the electronic
> communications markets. For competition to successfully discipline
> operators' behaviour in the interests of their customers, end user= s must
> be fully aware of the characteristics of the ISPs' access offers. = In some
> jurisdictions regulators mandate transparent information made availabl= e
> about service offers.
>
> End users need effective transparency to be able to make informed choi= ces
> throughout the different stages of their relationship with ISPs, when<= br> > selecting Internet access service offers, and when considering switchi= ng
> service offer within an ISP or to an alternative ISP. Quality informat= ion
> about service offers can be provided based on performance measurements= of
> the services provided in the market.
>
> A set of measurement parameters and associated measurement methods are=
> used over time, e.g. speed, delay, and jitter. Then the measurement ra= w
> data are collected and go through statistical post-processing before t= he
> results can be published in an Internet access service quality index t= o
> facilitate end users' choice of service provider and offer.
>
> In order to make information on offerings more meaningful and comparab= le
> to end users, common frames of reference on Internet access service > quality metrics are essential. An example would be that in relation to=
> access speed, the actual download and upload speeds are measured and > published, not only the maximum speed.
>
> A measurement system that monitor Internet access services and collect=
> quality information can typically consist of a number of measurement > probes and one or more test servers located at peering points. The sys= tem
> can be operated by a regulator or a measurement provider. =A0Number an= d
> distribution of probes follows specific requirements depending on the<= br> > scope and the desired statistical reliability of the measurement campa= ign.
>
> In many test configurations, the measurement results will only reflect= the
> performance of ISP's own network. However, the quality of an Inter= net
> access service is also determined by the connectivity to the rest of t= he
> Internet. Therefore test configurations which measure beyond the ISP&#= 39;s own
> network may also be relevant in some cases.
>
>
> 4.2 Promoting end user empowerment
> --------------------------------------------------
> In addition to the advantages that end users receive from the competit= ion
> in the market, through availability of a plethora of ISPs and Internet=
> access service offers, facilitated by transparent quality information<= br> > about these offers at a general level, end users would further leverag= e on
> information about specific services they subscribe to, or which they a= re
> considering to subscribe to.
>
> Performance of Internet access service offers vary based on where the<= br> > service is provided (geographical and topological aspects), how it is<= br> > provided (e.g. access technology and ISP's interconnection to the<= br> > Internet) and when it is provided (variation through the day and over = the
> week). Therefore there is a need to obtain more detailed quality
> information about specific service offers.
>
> Quality measurement tools can be made available for end users to monit= or
> their Internet access service, giving information about the speed and<= br> > other quality metrics of the subscribed service. Speed meters and simi= lar
> tools can be provided by regulators or other third parties, but also > directly by the ISP for the benefit of the subscriber.
>
> Such measurement tools assist end users in verifying the performance o= f
> the Internet access service provided, comparing the measurement result= s
> achieved with the information given in the contract. If the contracted=
> service quality level is not met, end users can use these results when=
> complaining to the ISP to achieve improvement and compensation.
>
> It may also help end users to assess whether they have chosen the pack= age
> which best suits their specific needs. With their performance results = at
> hand, they can benefit from quality evaluation guides, which offer
> comparisons based on their specific usage profiles. This may enable th= em
> to check whether their current subscription is the most appropriate, a= nd
> assist them to switch package or provider if it is in their best inter= est.
>
> When provided by regulators and other third parties, such quality
> measurement tools can also be used to compare performance between ISPs= . It
> should, though, be recognised that measurement results can be affected= by
> factors outside the ISP's control, related for instance with local= network
> connection (e.g. Wi-Fi), equipment and software being used.
>
>
> 4.3 Monitoring overall market development
> ---------------------------------------------------------
> Governments sometimes set strategic goals for high-speed Internet acce= ss
> penetration as an important component of the economic, cultural and so= cial
> development of the society. To evaluate the effect of the stimulated > growth over time, broadband Internet access take-up and penetration of=
> high-speed access can be monitored through measurement campaigns.
>
> An example of such an initiative is the "Digital Agenda for Europ= e" which
> was adopted in 2010, to achieve universal broadband access. The goal i= s to
> achieve by 2020, access for all Europeans to Internet access speeds of= 30
> Mbps or above, and 50% or more of European households subscribing to > Internet connections above 100 Mbps.
>
> To monitor actual broadband Internet access performance in a specific<= br> > country or a region, extensive measurement campaigns are needed. A pan= el
> can be built based on operators and packages in the market, spread ove= r
> urban, suburban and rural areas. Probes can then be distributed to the=
> participants of the campaign.
>
> Periodic tests running on the probes can for example measure actual sp= eed
> at peak and off-peak hours, but also other detailed quality metrics li= ke
> delay and jitter. Collected data goes afterwards through statistical > analysis, deriving estimates for the whole population which can then b= e
> presented and published regularly.
>
> Using a harmonized or standardised measurement methodology, or even a<= br> > common quality measurement platform, measurement results could also be=
> used for benchmarking of providers and/or countries.
>
>
> 4.4 Detecting degradation of the Internet access
> --------------------------------------------------------------
> Regulation related to net neutrality and the open Internet has been > introduced in some jurisdictions' Internet policy, and monitoring = methods
> for detection of potential net neutrality breaches is needed to follow= up
> such regulations with concrete actions.
>
>
> Net neutrality is typically related to equal treatment of traffic
> transmitted over the Internet access service, while other traffic
> transmitted in parallel over the end user's broadband connection e= .g. to
> provide IPTV in IP networks separated from the Internet (also referred= to
> as "closed IP networks") can be exempted from net neutrality= regulation.
> Services provided in parallel to the Internet access services are refe= rred
> to as specialised services (ref. BEREC and FCC).
>
> In order for such regulatory service architecture to protect net
> neutrality and the open Internet, it is essential that the specialised=
> services are not provided at the expense of the Internet access servic= e.
> Therefore regulators may need to measure performance of the Internet > access service over time, and thereby detect whether this service is > becoming degraded.
>
> When regulators monitor this development to ensure net neutrality and = the
> open Internet, they can check e.g. whether Internet access service
> performance and level of quality reflect advances in technology and th= at
> this is not impaired by specialised services. Comparison between ISPs = or
> between different countries may also be relevant for this kind of
> evaluation.
>
> Based on such Internet access service quality monitoring, regulators w= ill
> typically report results and findings regularly, e.g. on an annual bas= is.
> In some jurisdictions regulators may also eventually impose minimum > quality of service requirements on ISPs in order to prevent degradatio= n of
> Internet access services.
>
>
> 4.5 Detecting degradation of specific applications
> ---------------------------------------------------------------
> In the regulatory context related to net neutrality and the open Inter= net,
> in addition to the monitoring to detect potential degradation of the > access service as a whole described above, it is essential to perform<= br> > monitoring to detect potential degradation of individual applications<= br> > using the Internet access.
>
> Since net neutrality relates to equal treatment of traffic transmitted=
> over the Internet access service, the treatment of different applicati= ons'
> traffic is in the core of net neutrality. When traffic originating fro= m
> different applications are treated independent of which application ea= ch
> IP packet belongs to, this is referred to as application agnosticism.<= br> >
> Examples of departure from application agnosticism are blocking or
> throttling of traffic from specific applications, but also preferentia= l
> treatment of specific applications. If traffic is forwarded at differe= nt
> priority levels, traffic having higher priority level implies that oth= er
> traffic has lower priority. Detection of such traffic management pract= ices
> does not necessarily imply breaches to net neutrality, but these
> observations can be used as an input to the regulatory evaluation of t= he
> practice.
>
> When regulators monitor traffic management practices to ensure net
> neutrality and the open Internet, they can check compliance with crite= ria
> defining what is considered reasonable traffic management. Measurement=
> systems designed to monitor this kind of effects need to send applicat= ion-
> specific traffic and then measure in detail the behaviour of the diffe= rent
> packets receive when transferred over the Internet access service.
>
> Today there exist measurement tools that can detect differentiated
> treatment of individual applications. Another method that can be used = is
> to measure performance at application layer, e.g. assisted by content = and
> application providers, but also passive measurements are foreseen as a=
> promising approach for monitoring of application-specific treatment. >
> _______________________________________________
> lmap mailing list
> lmap@ietf.org
> https://www.ietf.org/mailman/listinfo/lmap
> _______________________________________________
> lmap mailing list
> lmap@ietf.org
> https://www.ietf.org/mailman/listinfo/lmap
_______________________________________________
lmap mailing list
lmap@ietf.org
ht= tps://www.ietf.org/mailman/listinfo/lmap
--001a11c23ed2e2453b04ea8328c6-- From Klaus.Nieminen@ficora.fi Wed Nov 6 07:30:25 2013 Return-Path: X-Original-To: lmap@ietfa.amsl.com Delivered-To: lmap@ietfa.amsl.com Received: from localhost (localhost [127.0.0.1]) by ietfa.amsl.com (Postfix) with ESMTP id 95C7A21E8107 for ; Wed, 6 Nov 2013 07:30:25 -0800 (PST) X-Virus-Scanned: amavisd-new at amsl.com X-Spam-Flag: NO X-Spam-Score: -2.599 X-Spam-Level: X-Spam-Status: No, score=-2.599 tagged_above=-999 required=5 tests=[BAYES_00=-2.599] Received: from mail.ietf.org ([12.22.58.30]) by localhost (ietfa.amsl.com [127.0.0.1]) (amavisd-new, port 10024) with ESMTP id AU6IP98oMIoN for ; Wed, 6 Nov 2013 07:30:20 -0800 (PST) Received: from out2.ficora.fi (out3.ficora.fi [87.239.126.62]) by ietfa.amsl.com (Postfix) with ESMTP id A35E011E810F for ; Wed, 6 Nov 2013 07:30:19 -0800 (PST) X-IronPort-AV: E=Sophos;i="4.93,647,1378846800"; d="scan'208";a="4950535" Received: from LOOTA.laru.local ([fe80::3cb5:71aa:ca40:2a8d]) by loota.laru.local ([fe80::3cb5:71aa:ca40:2a8d%10]) with mapi id 14.03.0158.001; Wed, 6 Nov 2013 17:30:14 +0200 From: Nieminen Klaus To: James Miller , "MORTON JR. ALFRED, (AL)" Thread-Topic: [lmap] LMAP regulator use case Thread-Index: AQHO2v59oBFaE8+BzkaRHmy3dVQTU5oYUWAQ Date: Wed, 6 Nov 2013 15:30:13 +0000 Message-ID: <6AF4522BD5AB86429CFD3948A9AB2F2F3A228999@loota.laru.local> References: <793D91975B99224DA9777562FF192808A27468CB@exmbx01> <2845723087023D4CB5114223779FA9C8ABF5C7D1@njfpsrvexg8.research.att.com> In-Reply-To: Accept-Language: en-US, fi-FI Content-Language: en-US X-MS-Has-Attach: X-MS-TNEF-Correlator: x-originating-ip: [87.239.126.79] Content-Type: text/plain; charset="Windows-1252" Content-Transfer-Encoding: quoted-printable MIME-Version: 1.0 Cc: "philip.eardley@bt.com" , "frode.sorensen@npt.no" , "lmap@ietf.org" Subject: Re: [lmap] LMAP regulator use case X-BeenThere: lmap@ietf.org X-Mailman-Version: 2.1.12 Precedence: list List-Id: Large Scale Measurement of Access network Performance List-Unsubscribe: , List-Archive: List-Post: List-Help: List-Subscribe: , X-List-Received-Date: Wed, 06 Nov 2013 15:30:25 -0000 Dear all, I would also like to add that as far as I have understood, LMAP WG don't st= andardize performance metrics. Therefore I think that we don't have to worr= y that much about how hard it is to measure degradation of the Internet acc= ess or specific applications. In addition as James and Frode mentioned ther= e are already some tools available for this purpose.=20 I fully support James' last remark that "it is an evolving space but someth= ing relevant for regulators use case description." - Klaus -----Original Message----- From: James Miller [mailto:jamesmilleresquire@gmail.com]=20 Sent: 6. marraskuuta 2013 16:43 To: MORTON JR. ALFRED, (AL) Cc: philip.eardley@bt.com; lmap@ietf.org; frode.sorensen@npt.no Subject: Re: [lmap] LMAP regulator use case I agree with Frode's comments on the 2.2 points. I would add the quality o= f the comparability of data is crucial for a regulators use and extends to = technical neutrality across the access link technologies as well, eg abilit= y to compare docsis, DSL, fiber, fixed wireless or other solutions. Most s= tudies I am familiar with include present and post processing techniques to= support comparisons across major broadband technologies that include out o= f scope topics. With 4.4 or 4.5 I agree there are a mix of views but would suggest there ar= e production quality tools as well as more research level quality topics. = I believe many are likely to extend to discussion of ippm metrics. In addi= tion to the tools mentioned, nick weaver's netalizer suite is an example th= at tests for easy things (like port blocking) and harder more subtle effect= s (like throttling, caching and proxy effects), and would also mention work= by bitag and other groups trying to reach technical consensus on descripti= on of many traffic management practices. It is an evolving space but somet= hing relevant for regulators use case description. On Nov 6, 2013 8:36 AM, "MORTON, ALFRED C (AL)" wrote: Hi Frode, =09 I read your mail and particularly agree with Phil's comments excerpted bel= ow: =09 > 4.4 Detecting degradation of the Internet access > 4.5 Detecting degradation of specific applications > > I don't like these sections about testing for whether the ISP is being n= et > neutral. > I think it would be very hard to design tests. > In part this is because net neutrality is more of a political concept th= an > technical one (in my view) > > I tihnk some of the stuff in S4.4 is really part of 4.3, as it's a marke= t > development issue (the internet has got slower, and doesnt meet the poli= cy > objective of Superfast broadband by Year 2015 - or whatever)- =09 If one form of the absence of neutrality is censorship, even censorship is hard to define and can include performance degradation according to Nick Feamster's presentation at the IRTF open meeting: http://www.ietf.org/proceedings/88/slides/slides-88-irtfopen-3.pptx =09 I think it's fair to say this sort of detection testing is still a researc= h topic, Al =09 > -----Original Message----- > From: lmap-bounces@ietf.org [mailto:lmap-bounces@ietf.org] On Behalf Of > philip.eardley@bt.com > Sent: Tuesday, November 05, 2013 9:42 PM > To: frode.sorensen@npt.no; lmap@ietf.org > Subject: Re: [lmap] LMAP regulator use case > > Frode > thanks very much for the comments - and it is excellent to have detailed > text suggestions. > > 2.2 > I think i'm basically OK with this replacing the current S2.2 (including > the subsections) > I suggest that it should mention comparability in the final para (as wel= l > as in the penultimate one) - personally I think this is the most importa= nt > characteristic from a regulator's point of view, so that it's fair to > compare measurements they make across the different ISPs. > the one thing in the current S2.2 (actually in S2.2.3) that I think is > worth keeping is a mention of fixed and mobile. I think this could be > handled by adding to the Intro (S1) something like: "Measurements will > take place on both fixed and mobile access" > > > Happy to add a new chapter 4, but comments about some of your sub use > cases > > 4.1 Promoting competition through transparency > >>For competition to successfully discipline operators' behaviour in the > interests of their customers, end users must be fully aware of the > characteristics of the ISPs' access offers. > "must be fully aware" over-states things. > > >>The system can be operated by a regulator or a measurement provider. > add "on behalf of the regulator" > > >> However, the quality of an Internet access service is also determine= d > by the connectivity to the rest of the Internet. Therefore test > configurations which measure beyond the ISP's own network may also be > relevant in some cases. > true. perhaps worth mentioning that the ISPs can influence this to some > extent - how well they're connected to the Internet, how they cache > popular content etc. > > 4.2 Promoting end user empowerment > i think this can be deleted. this is the end user use case, not the > regulator use case. > > 4.3 Monitoring overall market development > good for me > > 4.4 Detecting degradation of the Internet access > 4.5 Detecting degradation of specific applications > > I don't like these sections about testing for whether the ISP is being n= et > neutral. > I think it would be very hard to design tests. > In part this is because net neutrality is more of a political concept th= an > technical one (in my view) > > i tihnk some of the stuff in S4.4 is really part of 4.3, as it's a marke= t > development issue (the internet has got slower, and doesnt meet the poli= cy > objective of Superfast broadband by Year 2015 - or whatever)- > > best wishes > phil > > ________________________________________ > From: lmap-bounces@ietf.org [lmap-bounces@ietf.org] On Behalf Of S=F8ren= sen, > Frode [frode.sorensen@npt.no] > Sent: 30 October 2013 14:55 > To: lmap@ietf.org > Subject: [lmap] LMAP regulator use case > > Marc, Phil, Trevor, all, > > I have prepared a proposed description of a regulator use case for the > LMAP use case ID. I noticed the "advertisement" for further input from > regulators for the use case a couple of weeks ago. Finally I have manage= d > to find sufficient time to contribute to the ID, supported by a couple o= f > "regulator friends" from the Net Neutrality Working Group of the BEREC > organization, representing European national regulators. The proposal > consists of two parts; a modification of section 2.2 "Regulators", plus = a > new chapter after chapter 3 "Details of ISP use case", titled "Details o= f > regulator use case". > > Thanks, > Frode > > > Proposed update: > > 2.2 Regulators > ------------------- > Regulators in jurisdictions around the world are responding to consumers= ' > adoption of Internet access services for traditional telecommunications > and media services by promoting competition among providers of electroni= c > communications, to ensure that users derive maximum benefit in terms of > choice, price, and quality. > > Some jurisdictions have responded to a need for greater information abou= t > Internet access service performance in the development of regulatory > policies and approaches for broadband technologies by developing large- > scale measurement programs. Programs such as the U.S. Federal > Communications Commission's Measuring Broadband America, European > Commission's Quality of Broadband Services in the EU reports and a growi= ng > list of other programs employ a diverse set of operational and technical > approaches to gathering data to perform analysis and reporting on divers= e > aspects of broadband performance. > > While each jurisdiction responds to distinct consumer, industry, and > regulatory concerns, much commonality exists in the need to produce > datasets that are able to compare multiple Internet access service > providers, diverse technical solutions, geographic and regional > distributions, and marketed and provisioned levels and combinations of > broadband Internet access services. In some jurisdictions, the role of > measuring is provided by a measurement provider. > > Measurement providers measure network performance from users towards > multiple content and application providers, included dedicated test > measurement servers, to show a performance of the actual Internet access > service provided by different ISPs. Users need to know the performance > that are achieving from their own ISP. In addition, they need to know th= e > performance of other ISPs of same location as background information for > selecting their ISP. Measurement providers will provide measurement > results with associated measurement methods and measurement metrics. > > From a consumer perspective, the differentiation between fixed and mobil= e > (cellular) Internet access services is blurring as the applications used > are very similar. Hence, similar measurements will take place on both > fixed and mobile Internet access services. > > Regulators role in the development and enforcement of broadband Internet > access service policies also require that the measurement approaches mee= t > a high level of verifiability, accuracy and provider-independence to > support valid and meaningful comparisons of Internet access service > performance > > LMAP standards could answer regulators shared needs by providing scalabl= e, > cost-effective, scientifically robust solutions to the measurement and > collection of broadband Internet access service performance information. > > > Proposed new chapter: > > 4 Details of regulator use case > --------------------------------------- > 4.1 Promoting competition through transparency > --------------------------------------------------------------- > Competition plays a vital role in regulation of the electronic > communications markets. For competition to successfully discipline > operators' behaviour in the interests of their customers, end users must > be fully aware of the characteristics of the ISPs' access offers. In som= e > jurisdictions regulators mandate transparent information made available > about service offers. > > End users need effective transparency to be able to make informed choice= s > throughout the different stages of their relationship with ISPs, when > selecting Internet access service offers, and when considering switching > service offer within an ISP or to an alternative ISP. Quality informatio= n > about service offers can be provided based on performance measurements o= f > the services provided in the market. > > A set of measurement parameters and associated measurement methods are > used over time, e.g. speed, delay, and jitter. Then the measurement raw > data are collected and go through statistical post-processing before the > results can be published in an Internet access service quality index to > facilitate end users' choice of service provider and offer. > > In order to make information on offerings more meaningful and comparable > to end users, common frames of reference on Internet access service > quality metrics are essential. An example would be that in relation to > access speed, the actual download and upload speeds are measured and > published, not only the maximum speed. > > A measurement system that monitor Internet access services and collect > quality information can typically consist of a number of measurement > probes and one or more test servers located at peering points. The syste= m > can be operated by a regulator or a measurement provider. Number and > distribution of probes follows specific requirements depending on the > scope and the desired statistical reliability of the measurement campaig= n. > > In many test configurations, the measurement results will only reflect t= he > performance of ISP's own network. However, the quality of an Internet > access service is also determined by the connectivity to the rest of the > Internet. Therefore test configurations which measure beyond the ISP's o= wn > network may also be relevant in some cases. > > > 4.2 Promoting end user empowerment > -------------------------------------------------- > In addition to the advantages that end users receive from the competitio= n > in the market, through availability of a plethora of ISPs and Internet > access service offers, facilitated by transparent quality information > about these offers at a general level, end users would further leverage = on > information about specific services they subscribe to, or which they are > considering to subscribe to. > > Performance of Internet access service offers vary based on where the > service is provided (geographical and topological aspects), how it is > provided (e.g. access technology and ISP's interconnection to the > Internet) and when it is provided (variation through the day and over th= e > week). Therefore there is a need to obtain more detailed quality > information about specific service offers. > > Quality measurement tools can be made available for end users to monitor > their Internet access service, giving information about the speed and > other quality metrics of the subscribed service. Speed meters and simila= r > tools can be provided by regulators or other third parties, but also > directly by the ISP for the benefit of the subscriber. > > Such measurement tools assist end users in verifying the performance of > the Internet access service provided, comparing the measurement results > achieved with the information given in the contract. If the contracted > service quality level is not met, end users can use these results when > complaining to the ISP to achieve improvement and compensation. > > It may also help end users to assess whether they have chosen the packag= e > which best suits their specific needs. With their performance results at > hand, they can benefit from quality evaluation guides, which offer > comparisons based on their specific usage profiles. This may enable them > to check whether their current subscription is the most appropriate, and > assist them to switch package or provider if it is in their best interes= t. > > When provided by regulators and other third parties, such quality > measurement tools can also be used to compare performance between ISPs. = It > should, though, be recognised that measurement results can be affected b= y > factors outside the ISP's control, related for instance with local netwo= rk > connection (e.g. Wi-Fi), equipment and software being used. > > > 4.3 Monitoring overall market development > --------------------------------------------------------- > Governments sometimes set strategic goals for high-speed Internet access > penetration as an important component of the economic, cultural and soci= al > development of the society. To evaluate the effect of the stimulated > growth over time, broadband Internet access take-up and penetration of > high-speed access can be monitored through measurement campaigns. > > An example of such an initiative is the "Digital Agenda for Europe" whic= h > was adopted in 2010, to achieve universal broadband access. The goal is = to > achieve by 2020, access for all Europeans to Internet access speeds of 3= 0 > Mbps or above, and 50% or more of European households subscribing to > Internet connections above 100 Mbps. > > To monitor actual broadband Internet access performance in a specific > country or a region, extensive measurement campaigns are needed. A panel > can be built based on operators and packages in the market, spread over > urban, suburban and rural areas. Probes can then be distributed to the > participants of the campaign. > > Periodic tests running on the probes can for example measure actual spee= d > at peak and off-peak hours, but also other detailed quality metrics like > delay and jitter. Collected data goes afterwards through statistical > analysis, deriving estimates for the whole population which can then be > presented and published regularly. > > Using a harmonized or standardised measurement methodology, or even a > common quality measurement platform, measurement results could also be > used for benchmarking of providers and/or countries. > > > 4.4 Detecting degradation of the Internet access > -------------------------------------------------------------- > Regulation related to net neutrality and the open Internet has been > introduced in some jurisdictions' Internet policy, and monitoring method= s > for detection of potential net neutrality breaches is needed to follow u= p > such regulations with concrete actions. > > > Net neutrality is typically related to equal treatment of traffic > transmitted over the Internet access service, while other traffic > transmitted in parallel over the end user's broadband connection e.g. to > provide IPTV in IP networks separated from the Internet (also referred t= o > as "closed IP networks") can be exempted from net neutrality regulation. > Services provided in parallel to the Internet access services are referr= ed > to as specialised services (ref. BEREC and FCC). > > In order for such regulatory service architecture to protect net > neutrality and the open Internet, it is essential that the specialised > services are not provided at the expense of the Internet access service. > Therefore regulators may need to measure performance of the Internet > access service over time, and thereby detect whether this service is > becoming degraded. > > When regulators monitor this development to ensure net neutrality and th= e > open Internet, they can check e.g. whether Internet access service > performance and level of quality reflect advances in technology and that > this is not impaired by specialised services. Comparison between ISPs or > between different countries may also be relevant for this kind of > evaluation. > > Based on such Internet access service quality monitoring, regulators wil= l > typically report results and findings regularly, e.g. on an annual basis= . > In some jurisdictions regulators may also eventually impose minimum > quality of service requirements on ISPs in order to prevent degradation = of > Internet access services. > > > 4.5 Detecting degradation of specific applications > --------------------------------------------------------------- > In the regulatory context related to net neutrality and the open Interne= t, > in addition to the monitoring to detect potential degradation of the > access service as a whole described above, it is essential to perform > monitoring to detect potential degradation of individual applications > using the Internet access. > > Since net neutrality relates to equal treatment of traffic transmitted > over the Internet access service, the treatment of different application= s' > traffic is in the core of net neutrality. When traffic originating from > different applications are treated independent of which application each > IP packet belongs to, this is referred to as application agnosticism. > > Examples of departure from application agnosticism are blocking or > throttling of traffic from specific applications, but also preferential > treatment of specific applications. If traffic is forwarded at different > priority levels, traffic having higher priority level implies that other > traffic has lower priority. Detection of such traffic management practic= es > does not necessarily imply breaches to net neutrality, but these > observations can be used as an input to the regulatory evaluation of the > practice. > > When regulators monitor traffic management practices to ensure net > neutrality and the open Internet, they can check compliance with criteri= a > defining what is considered reasonable traffic management. Measurement > systems designed to monitor this kind of effects need to send applicatio= n- > specific traffic and then measure in detail the behaviour of the differe= nt > packets receive when transferred over the Internet access service. > > Today there exist measurement tools that can detect differentiated > treatment of individual applications. Another method that can be used is > to measure performance at application layer, e.g. assisted by content an= d > application providers, but also passive measurements are foreseen as a > promising approach for monitoring of application-specific treatment. > > _______________________________________________ > lmap mailing list > lmap@ietf.org > https://www.ietf.org/mailman/listinfo/lmap > _______________________________________________ > lmap mailing list > lmap@ietf.org > https://www.ietf.org/mailman/listinfo/lmap _______________________________________________ lmap mailing list lmap@ietf.org https://www.ietf.org/mailman/listinfo/lmap =09 From Michael.K.Bugenhagen@centurylink.com Wed Nov 6 09:03:51 2013 Return-Path: X-Original-To: lmap@ietfa.amsl.com Delivered-To: lmap@ietfa.amsl.com Received: from localhost (localhost [127.0.0.1]) by ietfa.amsl.com (Postfix) with ESMTP id 4919111E81FF for ; Wed, 6 Nov 2013 09:03:51 -0800 (PST) X-Virus-Scanned: amavisd-new at amsl.com X-Spam-Flag: NO X-Spam-Score: -2.599 X-Spam-Level: X-Spam-Status: No, score=-2.599 tagged_above=-999 required=5 tests=[AWL=-0.001, BAYES_00=-2.599, HTML_MESSAGE=0.001] Received: from mail.ietf.org ([12.22.58.30]) by localhost (ietfa.amsl.com [127.0.0.1]) (amavisd-new, port 10024) with ESMTP id mrqLPp-UK57x for ; Wed, 6 Nov 2013 09:03:45 -0800 (PST) Received: from suomp64i.qwest.com (suomp64i.qwest.com [155.70.16.237]) by ietfa.amsl.com (Postfix) with ESMTP id B8C3111E8140 for ; Wed, 6 Nov 2013 09:03:45 -0800 (PST) Received: from lxdenvmpc030.qintra.com (lxdenvmpc030.qintra.com [10.1.51.30]) by suomp64i.qwest.com (8.14.4/8.14.4) with ESMTP id rA6H3bOm004958 (version=TLSv1/SSLv3 cipher=DHE-RSA-AES256-SHA bits=256 verify=NO); Wed, 6 Nov 2013 11:03:37 -0600 (CST) Received: from lxdenvmpc030.qintra.com (unknown [127.0.0.1]) by IMSA (Postfix) with ESMTP id D07EA1E0061; Wed, 6 Nov 2013 10:03:31 -0700 (MST) Received: from suomp60i.qintra.com (unknown [151.119.91.93]) by lxdenvmpc030.qintra.com (Postfix) with ESMTP id 94A5A1E004E; Wed, 6 Nov 2013 10:03:31 -0700 (MST) Received: from suomp60i.qintra.com (localhost [127.0.0.1]) by suomp60i.qintra.com (8.14.4/8.14.4) with ESMTP id rA6H3V88020793; Wed, 6 Nov 2013 11:03:31 -0600 (CST) Received: from vodcwhubex501.ctl.intranet (vodcwhubex501.ctl.intranet [151.117.206.27]) by suomp60i.qintra.com (8.14.4/8.14.4) with ESMTP id rA6H3U3a020779 (version=TLSv1/SSLv3 cipher=AES128-SHA bits=128 verify=FAIL); Wed, 6 Nov 2013 11:03:30 -0600 (CST) Received: from PODCWMBXEX505.ctl.intranet ([fe80::f87e:fe44:ad72:b610]) by vodcwhubex501.ctl.intranet ([151.117.206.27]) with mapi id 14.02.0318.001; Wed, 6 Nov 2013 11:03:30 -0600 From: "Bugenhagen, Michael K" To: "'philip.eardley@bt.com'" , "'lmap@ietf.org'" Thread-Topic: 3 Suggested adds/ Edits for the LMAP Draft Thread-Index: Ac7bEh3r6TkvJxiSSsG+OjpDQgJCbA== Date: Wed, 6 Nov 2013 17:03:30 +0000 Message-ID: Accept-Language: en-US Content-Language: en-US X-MS-Has-Attach: X-MS-TNEF-Correlator: x-originating-ip: [151.117.206.7] Content-Type: multipart/alternative; boundary="_000_A68F3CAC468B2E48BB775ACE2DD99B5E04815118podcwmbxex505ct_" MIME-Version: 1.0 X-CFilter-Loop: Reflected Subject: [lmap] 3 Suggested adds/ Edits for the LMAP Draft X-BeenThere: lmap@ietf.org X-Mailman-Version: 2.1.12 Precedence: list List-Id: Large Scale Measurement of Access network Performance List-Unsubscribe: , List-Archive: List-Post: List-Help: List-Subscribe: , X-List-Received-Date: Wed, 06 Nov 2013 17:03:51 -0000 --_000_A68F3CAC468B2E48BB775ACE2DD99B5E04815118podcwmbxex505ct_ Content-Type: text/plain; charset="us-ascii" Content-Transfer-Encoding: quoted-printable Phil, Reviewing the draft http://www.ietf.org/id/draft-folks-lmap-framework-00= .txt Add's (in Red) Page 6 definitions - Consider adding .... 1) Add Measurement Agent testing protocol Measurement Agent Protocol: The control messages from one M= A to another used to conduct a test. 2) Add Service Attributes Service Attributes: The normative Service attribute values= that are pertinent to a test results during the timeframe the test was con= ducted. Examples - Sold speed - may equal the current service capaci= ty attribute for the timeframe the test is conducted. In use - Service state - under maintenance, down, up, .... Tweaks - Page 11 LMAP phases - suggested change for the "measurement task being perf= ormed" * MA Test Engine The MA execution of actual Measurement Tasks are performed. A MA evaluates running tasks based on environment constraints, and = service attributes, and runs test tasks Recording (logs and captures test results) the type of test completi= on state (successful, aborted, .....) It conducts Active Measurement Task involves sending test traffic be= tween the Measurement Agent and a Measurement Peer, whilst a Passive Measurement Task involves (only) the Measurement A= gent observing existing user traffic. The LMAP WG does not define Measurement Methods, however the IPPM WG does. Questions - 1) Lack of a Consumer / subscriber test point Discover Method - The draft states that "comparable" test points are used to allow co= mparability between test results. Is the assumption that ONLY the controller presents the test points= , or we the framework leverage something broader and more transparent such = as a DNS or other registry for test points, logically some points are= already DNS named, but it seems we are adding some that do not normally ge= t named, or we may what some ACL type restrictions to.... --_000_A68F3CAC468B2E48BB775ACE2DD99B5E04815118podcwmbxex505ct_ Content-Type: text/html; charset="us-ascii" Content-Transfer-Encoding: quoted-printable
Phil,
 
&nbs= p;
Add’s (in Red)
&nbs= p;
Page 6 definitions –  Consider adding ….
&nbs= p;
  1. Add Measurement Agent testing protocol
           = ;     Measurement Agent Protocol: The control messages from one MA to another used to conduct a test= .
 
  1. Add Service Attributes
    2.
           = ;     Service Attributes:  The normative Service a= ttribute values that are pertinent to a test results during the timeframe t= he test was conducted.
           = ;     Examples –
           = ;             = Sold speed - may equal the current service capacity attribute for the time= frame the test is conducted.
           = ;             = In use –
           = ;             S= ervice state – under maintenance, down, up, ….
 
Twea= ks –
Page= 11 LMAP phases – suggested change for the “measurement task be= ing performed”
&nbs= p;
  • MA Test Engine
  &n= bsp;     The MA execution of actual Measurement Tasks are performed. 
  &n= bsp;     A MA evaluates running tas= ks based on environment constraints, and service= attributes, and runs= test tasks
=    Recording (logs and captures test results) the type of test co= mpletion state (successful, aborted, …..)
=    It conducts Active Measurement Task invo= lves sending test traffic between the Measurement Agent and a Measurement P= eer,
        = whilst a Passive Measurement Task involves (only) the Measurement Agent obs= erving existing user traffic.  The
      LMAP WG does= not define Measurement Methods, however the IPPM WG
      does.=
 
&nbs= p;
Ques= tions –
&nbs= p;
  1. Lack of a Consumer / subscriber test point Discover Method –
    2. =
        The dra= ft states that “comparable” test points are used to allow compa= rability between test results.
        Is the = assumption that ONLY the controller presents the test points, or we the fra= mework leverage something broader and more transparent such as a DNS or oth= er registry for test        points, logical= ly some points are already DNS named, but it seems we are adding some that = do not normally get named, or we may what some ACL type restrictions toR= 30;. 
 
 
 
 
 --_000_A68F3CAC468B2E48BB775ACE2DD99B5E04815118podcwmbxex505ct_-- From dromasca@avaya.com Wed Nov 6 16:58:42 2013 Return-Path: X-Original-To: lmap@ietfa.amsl.com Delivered-To: lmap@ietfa.amsl.com Received: from localhost (localhost [127.0.0.1]) by ietfa.amsl.com (Postfix) with ESMTP id 9B8D121E8191 for ; Wed, 6 Nov 2013 16:58:42 -0800 (PST) X-Virus-Scanned: amavisd-new at amsl.com X-Spam-Flag: NO X-Spam-Score: -103.325 X-Spam-Level: X-Spam-Status: No, score=-103.325 tagged_above=-999 required=5 tests=[AWL=0.274, BAYES_00=-2.599, RCVD_IN_DNSWL_LOW=-1, USER_IN_WHITELIST=-100] Received: from mail.ietf.org ([12.22.58.30]) by localhost (ietfa.amsl.com [127.0.0.1]) (amavisd-new, port 10024) with ESMTP id cMGAB-gYsVu2 for ; Wed, 6 Nov 2013 16:58:33 -0800 (PST) Received: from de307622-de-outbound.net.avaya.com (de307622-de-outbound.net.avaya.com [198.152.71.100]) by ietfa.amsl.com (Postfix) with ESMTP id 5003111E816D for ; Wed, 6 Nov 2013 16:58:24 -0800 (PST) X-IronPort-Anti-Spam-Filtered: true X-IronPort-Anti-Spam-Result: AvoFAF3lelKHCzI1/2dsb2JhbABbgmYhgQu/LoEpFm0HgicBAQMSCx1RARUVFEImAQQbGodfAZ1HhEiceo4ggQiDWIEQA55wg2+HN4FogT6BaAcCFyI X-IronPort-AV: E=Sophos;i="4.93,647,1378872000"; d="scan'208";a="30697799" Received: from unknown (HELO p-us1-erheast.us1.avaya.com) ([135.11.50.53]) by de307622-de-outbound.net.avaya.com with ESMTP; 06 Nov 2013 19:58:19 -0500 Received: from unknown (HELO AZ-FFEXHC02.global.avaya.com) ([135.64.58.12]) by p-us1-erheast-out.us1.avaya.com with ESMTP; 06 Nov 2013 19:48:05 -0500 Received: from AZ-FFEXMB04.global.avaya.com ([fe80::6db7:b0af:8480:c126]) by AZ-FFEXHC02.global.avaya.com ([135.64.58.12]) with mapi id 14.03.0146.000; Thu, 7 Nov 2013 01:58:02 +0100 From: "Romascanu, Dan (Dan)" To: "lmap@ietf.org" Thread-Topic: call for consensus - adoption of draft-burbridge-lmap-information-model as WG document Thread-Index: Ac7bVGevh+4pI7FlSz+236qyIybWMQ== Date: Thu, 7 Nov 2013 00:58:01 +0000 Message-ID: <9904FB1B0159DA42B0B887B7FA8119CA1292EAA3@AZ-FFEXMB04.global.avaya.com> Accept-Language: en-US Content-Language: en-US X-MS-Has-Attach: X-MS-TNEF-Correlator: x-originating-ip: [135.64.58.45] Content-Type: text/plain; charset="us-ascii" Content-Transfer-Encoding: quoted-printable MIME-Version: 1.0 Subject: [lmap] call for consensus - adoption of draft-burbridge-lmap-information-model as WG document X-BeenThere: lmap@ietf.org X-Mailman-Version: 2.1.12 Precedence: list List-Id: Large Scale Measurement of Access network Performance List-Unsubscribe: , List-Archive: List-Post: List-Help: List-Subscribe: , X-List-Received-Date: Thu, 07 Nov 2013 00:58:42 -0000 Today in the LMAP meeting we ran a hum on the question whether we should ad= opt draft-burbridge-lmap-information-model. The result was read as showing = consensus in favor of the adoption, although there was at least one opposed= opinion expressed by Benoit (as contributor).=20 We would like to extend this call to the participants who could not attend = the meeting. Please send your opinions on this question to the WG mail list= before Friday 11/15.=20 Thanks and Regards, Dan From mailer@doodle.com Wed Nov 6 17:04:28 2013 Return-Path: X-Original-To: lmap@ietfa.amsl.com Delivered-To: lmap@ietfa.amsl.com Received: from localhost (localhost [127.0.0.1]) by ietfa.amsl.com (Postfix) with ESMTP id 265E421E81C5 for ; Wed, 6 Nov 2013 17:04:28 -0800 (PST) X-Virus-Scanned: amavisd-new at amsl.com X-Spam-Flag: NO X-Spam-Score: -11.854 X-Spam-Level: X-Spam-Status: No, score=-11.854 tagged_above=-999 required=5 tests=[AWL=-0.745, BAYES_05=-1.11, GB_I_INVITATION=-2, HTML_MESSAGE=0.001, RCVD_IN_DNSWL_HI=-8] Received: from mail.ietf.org ([12.22.58.30]) by localhost (ietfa.amsl.com [127.0.0.1]) (amavisd-new, port 10024) with ESMTP id ZpfBRoQJt6uK for ; Wed, 6 Nov 2013 17:04:23 -0800 (PST) Received: from worker2.doodle.com (worker2.doodle.com [188.92.145.82]) by ietfa.amsl.com (Postfix) with ESMTP id BEE3B11E816D for ; Wed, 6 Nov 2013 17:03:54 -0800 (PST) Received: from worker2.doodle.com (localhost [127.0.0.1]) by worker2.doodle.com (Postfix) with ESMTP id 6E97D9283E3 for ; Thu, 7 Nov 2013 02:03:52 +0100 (CET) Date: Thu, 7 Nov 2013 02:03:52 +0100 (CET) From: "Dan Romascanu (via Doodle)" To: Message-ID: <165158151.2730869.1383786232451.POLL_INVITECONTACT_PARTICIPANT_INVITATION.doodle@worker2> MIME-Version: 1.0 Content-Type: multipart/alternative; boundary="----=_Part_2730868_2121938840.1383786232450" Subject: [lmap] SACM WG Virtual Interim Meeting X-BeenThere: lmap@ietf.org X-Mailman-Version: 2.1.12 Precedence: list Reply-To: Dan Romascanu List-Id: Large Scale Measurement of Access network Performance List-Unsubscribe: , List-Archive: List-Post: List-Help: List-Subscribe: , X-List-Received-Date: Thu, 07 Nov 2013 01:04:28 -0000 ------=_Part_2730868_2121938840.1383786232450 Content-Type: text/plain; charset=UTF-8 Content-Transfer-Encoding: quoted-printable Hi there, Dan Romascanu (dromasca@avaya.com) invites you to participate in the Doodle poll "SACM WG Virtual Interim Meeting". Participate now https://doodle.com/vnwktt6msfpxavgz?tmail=3Dpoll_invitecontact_participant_= invitation&tlink=3Dpollbtn What is Doodle? Doodle is a web service that helps Dan Romascanu to find a suitable date for meeting with a group of people. Learn more about how Doodle works. (https://doodle.com/main.html?tlink=3DcheckOutLink&tmail=3Dpoll_inviteconta= ct_participant_invitation) ---------------------------------------------------------------------- You have received this e-mail because "Dan Romascanu" has invited you to participate in the Doodle poll "SACM WG Virtual Interim Meeting." ---- Doodle AG, Werdstrasse 21, 8021 Z=C3=BCrich ------=_Part_2730868_2121938840.1383786232450 Content-Type: text/html; charset=UTF-8 Content-Transfer-Encoding: quoted-printable =
=09
Dan Romascanu invites you t= o participate in the Doodle poll "SACM WG Virtual Interim Meeting.&quo= t;
=09
=09 =09 =09
=09 =09=09 =09=09 =09=09=09=09=09=09=09 =09=09 =09=09=09=09=09 =09=09=09=09=09=09=09 =09=09=09=09=09=09=09=09 =09 =09=09=09=09=09=09=09 =09=09=09=09=09=09=09=09 =09
=09=09 =09=09=09 =09=09 =09 =09=09 =09=09 =09=09 =09 =09=09 =09=09=09 =09=09 =09=09 =09=09 =09=09 =09 =09=09 =09=09
=09=09 =09 =09=09 =09 =09=09=09 =09 =09=09=09
=09
=09=09 =09=09=09 =09=09=09=09 =09=09=09=09=09
=09=09=09=09=09=09
=09=09 =09=09=09 =09=09=09=09 =09=09=09=09=09 =09=09=09=09=09=09=09 =09=09=09=09=09=09=09=09 =09=09=09=09=09=09=09 =09=09=09=09=09=09=09=09 =09 =09=09=09=09=09=09=09 =09=09=09=09=09=09=09=09 =09 =09=09=09=09=09=09=09 =09=09=09=09=09=09=09=09 =09 =09=09=09=09=09=09=09 =09=09=09=09=09=09=09=09 =09 =09 =09=09
=09=09=09=09=09=09
=09=09 =09=09 =09=09=09 =09=09=09=09 =09=09=09=09=09 =09=09=09=09 =09=09=09 =09=09
=09=09=09=09=09=09
"SACM WG Virt= ual Interim Meeting". =09=09=09=09=09=09
=09=09=09=09=09
=09=09 =09
=09=09 =09=09=09 =09=09=09=09 =09=09=09=09=09 =09=09=09=09 =09=09=09=09 =09=09=09=09=09 =09=09=09=09 =09=09=09=09 =09=09=09=09=09 =09=09=09=09 =09=09=09 =09=09
=
=09=09=09=09=09=09 =09=09=09=09=09=09 =09=09=09=09=09=09 =09=09=09=09=09=09=09 =09=09=09=09=09=09 =09=09=09=09=09=09 =09=09=09=09=09=09 =09=09=09=09=09
=09=09=09=09=09=09=09=09Participate now =09=09=09=09=09=09=09
=09=09=09=09=09
=
=09
=09=09 =09=09=09 =09=09=09=09 =09=09=09=09=09 =09=09=09=09=09 =09=09=09=09 =09=09=09 =09=09
=09=09=09=09=09=09 =09=09=09=09=09 =09=09=09=09=09=09
=20 =09=09=09=09=09=09=09 What is Doodle? Doodle is a web service that helps Dan Romascanu to find a suitable date= for meeting with a group of people. Learn more about how Doodle works.
=20 =09=09=09=09=09=09
=09=09=09=09=09
=09
=09=09 =09=09 =09 =09=09 =09=09 =09=09 =09 =09=09 =09 =09=09 =09 =09=09 =09 =09=09 =09=09 =09=09 =09 =09=09 =09=09 =20 =09=09
=09=09 =09You have received this e-mail because "Dan Romascanu&= quot; has invited you to participate in the Doodle poll "SACM WG Virtu= al Interim Meeting." =09=09
=09
=09 =09=09 =09=09=09 =09=09=09=09 =09=09=09=09=09
=09=09=09=09=09=09
To: "Romascanu, Dan (Dan)" , "lmap@ietf.org" Thread-Topic: [lmap] SACM WG Virtual Interim Meeting Thread-Index: AQHO21VgU2rGQfZLRk2CBMHgGMui85oY9IEQ Date: Thu, 7 Nov 2013 01:07:43 +0000 Message-ID: <9904FB1B0159DA42B0B887B7FA8119CA1292FAF4@AZ-FFEXMB04.global.avaya.com> References: <165158151.2730869.1383786232451.POLL_INVITECONTACT_PARTICIPANT_INVITATION.doodle@worker2> In-Reply-To: <165158151.2730869.1383786232451.POLL_INVITECONTACT_PARTICIPANT_INVITATION.doodle@worker2> Accept-Language: en-US Content-Language: en-US X-MS-Has-Attach: yes X-MS-TNEF-Correlator: x-originating-ip: [135.64.58.45] Content-Type: multipart/related; boundary="_005_9904FB1B0159DA42B0B887B7FA8119CA1292FAF4AZFFEXMB04globa_"; type="multipart/alternative" MIME-Version: 1.0 Subject: Re: [lmap] SACM WG Virtual Interim Meeting X-BeenThere: lmap@ietf.org X-Mailman-Version: 2.1.12 Precedence: list List-Id: Large Scale Measurement of Access network Performance List-Unsubscribe: , List-Archive: List-Post: List-Help: List-Subscribe: , X-List-Received-Date: Thu, 07 Nov 2013 01:08:36 -0000 --_005_9904FB1B0159DA42B0B887B7FA8119CA1292FAF4AZFFEXMB04globa_ Content-Type: multipart/alternative; boundary="_000_9904FB1B0159DA42B0B887B7FA8119CA1292FAF4AZFFEXMB04globa_" --_000_9904FB1B0159DA42B0B887B7FA8119CA1292FAF4AZFFEXMB04globa_ Content-Type: text/plain; charset="utf-8" Content-Transfer-Encoding: base64 TE1BUCBwYXJ0aWNpcGFudHMsDQoNClBsZWFzZSBkaXNyZWdhcmQgdGhpcyBtZXNzYWdlLiBJdCB3 YXMgbm90IG1lYW50IHRvIGJlIHNlbnQgdG8gdGhlIExNQVAgV0cuIEJsYW1lIGpldC1sYWchDQoN ClJlZ2FyZHMsDQoNCkRhbg0KDQoNCg0KRnJvbTogbG1hcC1ib3VuY2VzQGlldGYub3JnIFttYWls dG86bG1hcC1ib3VuY2VzQGlldGYub3JnXSBPbiBCZWhhbGYgT2YgRGFuIFJvbWFzY2FudSAodmlh IERvb2RsZSkNClNlbnQ6IFRodXJzZGF5LCBOb3ZlbWJlciAwNywgMjAxMyAzOjA0IEFNDQpUbzog bG1hcEBpZXRmLm9yZw0KU3ViamVjdDogW2xtYXBdIFNBQ00gV0cgVmlydHVhbCBJbnRlcmltIE1l ZXRpbmcNCg0KRGFuIFJvbWFzY2FudSBpbnZpdGVzIHlvdSB0byBwYXJ0aWNpcGF0ZSBpbiB0aGUg RG9vZGxlIHBvbGwgIlNBQ00gV0cgVmlydHVhbCBJbnRlcmltIE1lZXRpbmcuIg0KDQoNCg0KW0lt YWdlIHJlbW92ZWQgYnkgc2VuZGVyLl08aHR0cHM6Ly9kb29kbGUuY29tLz90bWFpbD1wb2xsX2lu dml0ZWNvbnRhY3RfcGFydGljaXBhbnRfaW52aXRhdGlvbiZ0bGluaz1sb2dvPg0KDQoNCg0KSGkg dGhlcmUsDQoNCg0KDQoNCg0KRGFuIFJvbWFzY2FudSAoZHJvbWFzY2FAYXZheWEuY29tKSBpbnZp dGVzIHlvdSB0byBwYXJ0aWNpcGF0ZSBpbiB0aGUgRG9vZGxlIHBvbGwgIlNBQ00gV0cgVmlydHVh bCBJbnRlcmltIE1lZXRpbmciLg0KDQoNCg0KUGFydGljaXBhdGUgbm93PGh0dHBzOi8vZG9vZGxl LmNvbS92bndrdHQ2bXNmcHhhdmd6P3RtYWlsPXBvbGxfaW52aXRlY29udGFjdF9wYXJ0aWNpcGFu dF9pbnZpdGF0aW9uJnRsaW5rPXBvbGxidG4+DQoNCg0KDQoNCltJbWFnZSByZW1vdmVkIGJ5IHNl bmRlci5dDQoNCldoYXQgaXMgRG9vZGxlPyBEb29kbGUgaXMgYSB3ZWIgc2VydmljZSB0aGF0IGhl bHBzIERhbiBSb21hc2NhbnUgdG8gZmluZCBhIHN1aXRhYmxlIGRhdGUgZm9yIG1lZXRpbmcgd2l0 aCBhIGdyb3VwIG9mIHBlb3BsZS4gTGVhcm4gbW9yZSBhYm91dCBob3cgRG9vZGxlIHdvcmtzLjxo dHRwczovL2Rvb2RsZS5jb20vbWFpbi5odG1sP3RsaW5rPWNoZWNrT3V0TGluayZ0bWFpbD1wb2xs X2ludml0ZWNvbnRhY3RfcGFydGljaXBhbnRfaW52aXRhdGlvbj4NCg0KDQoNCllvdSBoYXZlIHJl Y2VpdmVkIHRoaXMgZS1tYWlsIGJlY2F1c2UgIkRhbiBSb21hc2NhbnUiIGhhcyBpbnZpdGVkIHlv dSB0byBwYXJ0aWNpcGF0ZSBpbiB0aGUgRG9vZGxlIHBvbGwgIlNBQ00gV0cgVmlydHVhbCBJbnRl cmltIE1lZXRpbmcuIg0KDQoNCg0KRG9vZGxlIEFHLCBXZXJkc3RyYXNzZSAyMSwgODAyMSBaw7xy aWNoDQoNCg0KDQoNCg== --_000_9904FB1B0159DA42B0B887B7FA8119CA1292FAF4AZFFEXMB04globa_ Content-Type: text/html; charset="utf-8" Content-Transfer-Encoding: base64 PGh0bWwgeG1sbnM6dj0idXJuOnNjaGVtYXMtbWljcm9zb2Z0LWNvbTp2bWwiIHhtbG5zOm89InVy bjpzY2hlbWFzLW1pY3Jvc29mdC1jb206b2ZmaWNlOm9mZmljZSIgeG1sbnM6dz0idXJuOnNjaGVt YXMtbWljcm9zb2Z0LWNvbTpvZmZpY2U6d29yZCIgeG1sbnM6bT0iaHR0cDovL3NjaGVtYXMubWlj cm9zb2Z0LmNvbS9vZmZpY2UvMjAwNC8xMi9vbW1sIiB4bWxucz0iaHR0cDovL3d3dy53My5vcmcv VFIvUkVDLWh0bWw0MCI+DQo8aGVhZD4NCjxtZXRhIGh0dHAtZXF1aXY9IkNvbnRlbnQtVHlwZSIg Y29udGVudD0idGV4dC9odG1sOyBjaGFyc2V0PXV0Zi04Ij4NCjxtZXRhIG5hbWU9IkdlbmVyYXRv ciIgY29udGVudD0iTWljcm9zb2Z0IFdvcmQgMTIgKGZpbHRlcmVkIG1lZGl1bSkiPg0KPCEtLVtp ZiAhbXNvXT48c3R5bGU+dlw6KiB7YmVoYXZpb3I6dXJsKCNkZWZhdWx0I1ZNTCk7fQ0Kb1w6KiB7 YmVoYXZpb3I6dXJsKCNkZWZhdWx0I1ZNTCk7fQ0Kd1w6KiB7YmVoYXZpb3I6dXJsKCNkZWZhdWx0 I1ZNTCk7fQ0KLnNoYXBlIHtiZWhhdmlvcjp1cmwoI2RlZmF1bHQjVk1MKTt9DQo8L3N0eWxlPjwh W2VuZGlmXS0tPjxzdHlsZT48IS0tDQovKiBGb250IERlZmluaXRpb25zICovDQpAZm9udC1mYWNl DQoJe2ZvbnQtZmFtaWx5OkNhbGlicmk7DQoJcGFub3NlLTE6MiAxNSA1IDIgMiAyIDQgMyAyIDQ7 fQ0KQGZvbnQtZmFjZQ0KCXtmb250LWZhbWlseTpUYWhvbWE7DQoJcGFub3NlLTE6MiAxMSA2IDQg MyA1IDQgNCAyIDQ7fQ0KLyogU3R5bGUgRGVmaW5pdGlvbnMgKi8NCnAuTXNvTm9ybWFsLCBsaS5N c29Ob3JtYWwsIGRpdi5Nc29Ob3JtYWwNCgl7bWFyZ2luOjBpbjsNCgltYXJnaW4tYm90dG9tOi4w MDAxcHQ7DQoJZm9udC1zaXplOjEyLjBwdDsNCglmb250LWZhbWlseToiVGltZXMgTmV3IFJvbWFu Iiwic2VyaWYiO30NCmE6bGluaywgc3Bhbi5Nc29IeXBlcmxpbmsNCgl7bXNvLXN0eWxlLXByaW9y aXR5Ojk5Ow0KCWNvbG9yOmJsdWU7DQoJdGV4dC1kZWNvcmF0aW9uOnVuZGVybGluZTt9DQphOnZp c2l0ZWQsIHNwYW4uTXNvSHlwZXJsaW5rRm9sbG93ZWQNCgl7bXNvLXN0eWxlLXByaW9yaXR5Ojk5 Ow0KCWNvbG9yOnB1cnBsZTsNCgl0ZXh0LWRlY29yYXRpb246dW5kZXJsaW5lO30NCnNwYW4uRW1h aWxTdHlsZTE3DQoJe21zby1zdHlsZS10eXBlOnBlcnNvbmFsLXJlcGx5Ow0KCWZvbnQtZmFtaWx5 OiJDYWxpYnJpIiwic2Fucy1zZXJpZiI7DQoJY29sb3I6IzFGNDk3RDt9DQouTXNvQ2hwRGVmYXVs dA0KCXttc28tc3R5bGUtdHlwZTpleHBvcnQtb25seTsNCglmb250LXNpemU6MTAuMHB0O30NCkBw YWdlIFdvcmRTZWN0aW9uMQ0KCXtzaXplOjguNWluIDExLjBpbjsNCgltYXJnaW46MS4waW4gMS4y NWluIDEuMGluIDEuMjVpbjt9DQpkaXYuV29yZFNlY3Rpb24xDQoJe3BhZ2U6V29yZFNlY3Rpb24x O30NCi0tPjwvc3R5bGU+PCEtLVtpZiBndGUgbXNvIDldPjx4bWw+DQo8bzpzaGFwZWRlZmF1bHRz IHY6ZXh0PSJlZGl0IiBzcGlkbWF4PSIxMDI2IiAvPg0KPC94bWw+PCFbZW5kaWZdLS0+PCEtLVtp ZiBndGUgbXNvIDldPjx4bWw+DQo8bzpzaGFwZWxheW91dCB2OmV4dD0iZWRpdCI+DQo8bzppZG1h cCB2OmV4dD0iZWRpdCIgZGF0YT0iMSIgLz4NCjwvbzpzaGFwZWxheW91dD48L3htbD48IVtlbmRp Zl0tLT4NCjwvaGVhZD4NCjxib2R5IGxhbmc9IkVOLVVTIiBsaW5rPSJibHVlIiB2bGluaz0icHVy cGxlIj4NCjxkaXYgY2xhc3M9IldvcmRTZWN0aW9uMSI+DQo8cCBjbGFzcz0iTXNvTm9ybWFsIj48 c3BhbiBzdHlsZT0iZm9udC1zaXplOjExLjBwdDtmb250LWZhbWlseTomcXVvdDtDYWxpYnJpJnF1 b3Q7LCZxdW90O3NhbnMtc2VyaWYmcXVvdDs7Y29sb3I6IzFGNDk3RCI+TE1BUCBwYXJ0aWNpcGFu dHMsPG86cD48L286cD48L3NwYW4+PC9wPg0KPHAgY2xhc3M9Ik1zb05vcm1hbCI+PHNwYW4gc3R5 bGU9ImZvbnQtc2l6ZToxMS4wcHQ7Zm9udC1mYW1pbHk6JnF1b3Q7Q2FsaWJyaSZxdW90OywmcXVv dDtzYW5zLXNlcmlmJnF1b3Q7O2NvbG9yOiMxRjQ5N0QiPjxvOnA+Jm5ic3A7PC9vOnA+PC9zcGFu PjwvcD4NCjxwIGNsYXNzPSJNc29Ob3JtYWwiPjxzcGFuIHN0eWxlPSJmb250LXNpemU6MTEuMHB0 O2ZvbnQtZmFtaWx5OiZxdW90O0NhbGlicmkmcXVvdDssJnF1b3Q7c2Fucy1zZXJpZiZxdW90Oztj b2xvcjojMUY0OTdEIj5QbGVhc2UgZGlzcmVnYXJkIHRoaXMgbWVzc2FnZS4gSXQgd2FzIG5vdCBt ZWFudCB0byBiZSBzZW50IHRvIHRoZSBMTUFQIFdHLiBCbGFtZSBqZXQtbGFnITxvOnA+PC9vOnA+ PC9zcGFuPjwvcD4NCjxwIGNsYXNzPSJNc29Ob3JtYWwiPjxzcGFuIHN0eWxlPSJmb250LXNpemU6 MTEuMHB0O2ZvbnQtZmFtaWx5OiZxdW90O0NhbGlicmkmcXVvdDssJnF1b3Q7c2Fucy1zZXJpZiZx dW90Oztjb2xvcjojMUY0OTdEIj48bzpwPiZuYnNwOzwvbzpwPjwvc3Bhbj48L3A+DQo8cCBjbGFz cz0iTXNvTm9ybWFsIj48c3BhbiBzdHlsZT0iZm9udC1zaXplOjExLjBwdDtmb250LWZhbWlseTom cXVvdDtDYWxpYnJpJnF1b3Q7LCZxdW90O3NhbnMtc2VyaWYmcXVvdDs7Y29sb3I6IzFGNDk3RCI+ UmVnYXJkcyw8bzpwPjwvbzpwPjwvc3Bhbj48L3A+DQo8cCBjbGFzcz0iTXNvTm9ybWFsIj48c3Bh biBzdHlsZT0iZm9udC1zaXplOjExLjBwdDtmb250LWZhbWlseTomcXVvdDtDYWxpYnJpJnF1b3Q7 LCZxdW90O3NhbnMtc2VyaWYmcXVvdDs7Y29sb3I6IzFGNDk3RCI+PG86cD4mbmJzcDs8L286cD48 L3NwYW4+PC9wPg0KPHAgY2xhc3M9Ik1zb05vcm1hbCI+PHNwYW4gc3R5bGU9ImZvbnQtc2l6ZTox MS4wcHQ7Zm9udC1mYW1pbHk6JnF1b3Q7Q2FsaWJyaSZxdW90OywmcXVvdDtzYW5zLXNlcmlmJnF1 b3Q7O2NvbG9yOiMxRjQ5N0QiPkRhbjxvOnA+PC9vOnA+PC9zcGFuPjwvcD4NCjxwIGNsYXNzPSJN c29Ob3JtYWwiPjxzcGFuIHN0eWxlPSJmb250LXNpemU6MTEuMHB0O2ZvbnQtZmFtaWx5OiZxdW90 O0NhbGlicmkmcXVvdDssJnF1b3Q7c2Fucy1zZXJpZiZxdW90Oztjb2xvcjojMUY0OTdEIj48bzpw PiZuYnNwOzwvbzpwPjwvc3Bhbj48L3A+DQo8cCBjbGFzcz0iTXNvTm9ybWFsIj48c3BhbiBzdHls ZT0iZm9udC1zaXplOjExLjBwdDtmb250LWZhbWlseTomcXVvdDtDYWxpYnJpJnF1b3Q7LCZxdW90 O3NhbnMtc2VyaWYmcXVvdDs7Y29sb3I6IzFGNDk3RCI+PG86cD4mbmJzcDs8L286cD48L3NwYW4+ PC9wPg0KPHAgY2xhc3M9Ik1zb05vcm1hbCI+PHNwYW4gc3R5bGU9ImZvbnQtc2l6ZToxMS4wcHQ7 Zm9udC1mYW1pbHk6JnF1b3Q7Q2FsaWJyaSZxdW90OywmcXVvdDtzYW5zLXNlcmlmJnF1b3Q7O2Nv bG9yOiMxRjQ5N0QiPjxvOnA+Jm5ic3A7PC9vOnA+PC9zcGFuPjwvcD4NCjxkaXYgc3R5bGU9ImJv cmRlcjpub25lO2JvcmRlci1sZWZ0OnNvbGlkIGJsdWUgMS41cHQ7cGFkZGluZzowaW4gMGluIDBp biA0LjBwdCI+DQo8ZGl2Pg0KPGRpdiBzdHlsZT0iYm9yZGVyOm5vbmU7Ym9yZGVyLXRvcDpzb2xp ZCAjQjVDNERGIDEuMHB0O3BhZGRpbmc6My4wcHQgMGluIDBpbiAwaW4iPg0KPHAgY2xhc3M9Ik1z b05vcm1hbCI+PGI+PHNwYW4gc3R5bGU9ImZvbnQtc2l6ZToxMC4wcHQ7Zm9udC1mYW1pbHk6JnF1 b3Q7VGFob21hJnF1b3Q7LCZxdW90O3NhbnMtc2VyaWYmcXVvdDsiPkZyb206PC9zcGFuPjwvYj48 c3BhbiBzdHlsZT0iZm9udC1zaXplOjEwLjBwdDtmb250LWZhbWlseTomcXVvdDtUYWhvbWEmcXVv dDssJnF1b3Q7c2Fucy1zZXJpZiZxdW90OyI+IGxtYXAtYm91bmNlc0BpZXRmLm9yZyBbbWFpbHRv OmxtYXAtYm91bmNlc0BpZXRmLm9yZ10NCjxiPk9uIEJlaGFsZiBPZiA8L2I+RGFuIFJvbWFzY2Fu dSAodmlhIERvb2RsZSk8YnI+DQo8Yj5TZW50OjwvYj4gVGh1cnNkYXksIE5vdmVtYmVyIDA3LCAy MDEzIDM6MDQgQU08YnI+DQo8Yj5Ubzo8L2I+IGxtYXBAaWV0Zi5vcmc8YnI+DQo8Yj5TdWJqZWN0 OjwvYj4gW2xtYXBdIFNBQ00gV0cgVmlydHVhbCBJbnRlcmltIE1lZXRpbmc8bzpwPjwvbzpwPjwv c3Bhbj48L3A+DQo8L2Rpdj4NCjwvZGl2Pg0KPHAgY2xhc3M9Ik1zb05vcm1hbCI+PG86cD4mbmJz cDs8L286cD48L3A+DQo8ZGl2Pg0KPGRpdj4NCjxwIGNsYXNzPSJNc29Ob3JtYWwiIHN0eWxlPSJi YWNrZ3JvdW5kOiNGREZERkQiPkRhbiBSb21hc2NhbnUgaW52aXRlcyB5b3UgdG8gcGFydGljaXBh dGUgaW4gdGhlIERvb2RsZSBwb2xsICZxdW90O1NBQ00gV0cgVmlydHVhbCBJbnRlcmltIE1lZXRp bmcuJnF1b3Q7PG86cD48L286cD48L3A+DQo8L2Rpdj4NCjxkaXYgYWxpZ249ImNlbnRlciI+DQo8 dGFibGUgY2xhc3M9Ik1zb05vcm1hbFRhYmxlIiBib3JkZXI9IjAiIGNlbGxzcGFjaW5nPSIwIiBj ZWxscGFkZGluZz0iMCIgd2lkdGg9IjEwMCUiIHN0eWxlPSJ3aWR0aDoxMDAuMCU7YmFja2dyb3Vu ZDojRkRGREZEO2hlaWdodDoxMDAlIWltcG9ydGFudDt3aWR0aDoxMDAlIWltcG9ydGFudCI+DQo8 dGJvZHk+DQo8dHI+DQo8dGQgdmFsaWduPSJ0b3AiIHN0eWxlPSJwYWRkaW5nOjBpbiAwaW4gMGlu IDBpbiI+DQo8ZGl2IGFsaWduPSJjZW50ZXIiPg0KPHRhYmxlIGNsYXNzPSJNc29Ob3JtYWxUYWJs ZSIgYm9yZGVyPSIwIiBjZWxsc3BhY2luZz0iMCIgY2VsbHBhZGRpbmc9IjAiIHdpZHRoPSI0ODAi IHN0eWxlPSJ3aWR0aDo1LjBpbjtiYWNrZ3JvdW5kOiNGREZERkQiPg0KPHRib2R5Pg0KPHRyIHN0 eWxlPSJoZWlnaHQ6MjEuMHB0Ij4NCjx0ZCBjb2xzcGFuPSI0IiBzdHlsZT0icGFkZGluZzowaW4g MGluIDBpbiAwaW47aGVpZ2h0OjIxLjBwdCI+PC90ZD4NCjwvdHI+DQo8dHI+DQo8dGQgc3R5bGU9 InBhZGRpbmc6MGluIDBpbiAwaW4gMGluIj4NCjx0YWJsZSBjbGFzcz0iTXNvTm9ybWFsVGFibGUi IGJvcmRlcj0iMCIgY2VsbHNwYWNpbmc9IjAiIGNlbGxwYWRkaW5nPSIwIiB3aWR0aD0iNDgwIiBz dHlsZT0id2lkdGg6NS4waW47YmFja2dyb3VuZDp3aGl0ZSIgaWQ9InRlbXBsYXRlSGVhZGVyIj4N Cjx0Ym9keT4NCjx0ciBzdHlsZT0iaGVpZ2h0OjIxLjBwdCI+DQo8dGQgY29sc3Bhbj0iNCIgc3R5 bGU9InBhZGRpbmc6MGluIDBpbiAwaW4gMGluO2hlaWdodDoyMS4wcHQiPjwvdGQ+DQo8L3RyPg0K PHRyIHN0eWxlPSJoZWlnaHQ6MTkuNXB0Ij4NCjx0ZCB3aWR0aD0iMTUiIHZhbGlnbj0iYm90dG9t IiBzdHlsZT0id2lkdGg6MTEuMjVwdDtwYWRkaW5nOjBpbiAwaW4gMGluIDBpbjtoZWlnaHQ6MTku NXB0Ij4NCjwvdGQ+DQo8dGQgd2lkdGg9IjEyMCIgdmFsaWduPSJib3R0b20iIHN0eWxlPSJ3aWR0 aDoxLjI1aW47cGFkZGluZzowaW4gMGluIDBpbiAwaW47aGVpZ2h0OjE5LjVwdCI+DQo8cCBjbGFz cz0iTXNvTm9ybWFsIj48YSBocmVmPSJodHRwczovL2Rvb2RsZS5jb20vP3RtYWlsPXBvbGxfaW52 aXRlY29udGFjdF9wYXJ0aWNpcGFudF9pbnZpdGF0aW9uJmFtcDt0bGluaz1sb2dvIj48c3BhbiBz dHlsZT0iYm9yZGVyOnNvbGlkIHdpbmRvd3RleHQgMS4wcHQ7cGFkZGluZzowaW47dGV4dC1kZWNv cmF0aW9uOm5vbmUiPjxpbWcgYm9yZGVyPSIwIiB3aWR0aD0iMjYiIGhlaWdodD0iMjYiIGlkPSJf eDAwMDBfaTEwMjUiIHNyYz0iY2lkOmltYWdlMDAxLmpwZ0AwMUNFREI2Ni44MEM4RTY0MCIgYWx0 PSJJbWFnZSByZW1vdmVkIGJ5IHNlbmRlci4iPjwvc3Bhbj48L2E+PG86cD48L286cD48L3A+DQo8 L3RkPg0KPHRkIHdpZHRoPSIzMzAiIHZhbGlnbj0iYm90dG9tIiBzdHlsZT0id2lkdGg6MjQ3LjVw dDtwYWRkaW5nOjBpbiAwaW4gMGluIDBpbjtoZWlnaHQ6MTkuNXB0Ij4NCjwvdGQ+DQo8dGQgd2lk dGg9IjE1IiB2YWxpZ249ImJvdHRvbSIgc3R5bGU9IndpZHRoOjExLjI1cHQ7cGFkZGluZzowaW4g MGluIDBpbiAwaW47aGVpZ2h0OjE5LjVwdCI+DQo8L3RkPg0KPC90cj4NCjx0ciBzdHlsZT0iaGVp Z2h0OjkuMHB0Ij4NCjx0ZCBjb2xzcGFuPSI0IiBzdHlsZT0icGFkZGluZzowaW4gMGluIDBpbiAw aW47aGVpZ2h0OjkuMHB0Ij48L3RkPg0KPC90cj4NCjwvdGJvZHk+DQo8L3RhYmxlPg0KPC90ZD4N Cjx0ZCBzdHlsZT0icGFkZGluZzowaW4gMGluIDBpbiAwaW4iPjwvdGQ+DQo8dGQgc3R5bGU9InBh ZGRpbmc6MGluIDBpbiAwaW4gMGluIj48L3RkPg0KPHRkIHN0eWxlPSJwYWRkaW5nOjBpbiAwaW4g MGluIDBpbiI+PC90ZD4NCjwvdHI+DQo8dHI+DQo8dGQgdmFsaWduPSJ0b3AiIHN0eWxlPSJib3Jk ZXI6bm9uZTtib3JkZXItdG9wOnNvbGlkICNFMEU3RjAgMS4wcHQ7YmFja2dyb3VuZDojRjVGOUZE O3BhZGRpbmc6MGluIDBpbiAwaW4gMGluIj4NCjx0YWJsZSBjbGFzcz0iTXNvTm9ybWFsVGFibGUi IGJvcmRlcj0iMCIgY2VsbHNwYWNpbmc9IjAiIGNlbGxwYWRkaW5nPSIwIiB3aWR0aD0iMTAwJSIg c3R5bGU9IndpZHRoOjEwMC4wJSI+DQo8dGJvZHk+DQo8dHI+DQo8dGQgdmFsaWduPSJ0b3AiIHN0 eWxlPSJwYWRkaW5nOjE1LjBwdCAxMS4yNXB0IDBpbiAxMS4yNXB0Ij4NCjxwIGNsYXNzPSJNc29O b3JtYWwiIHN0eWxlPSJsaW5lLWhlaWdodDoxOC43NXB0Ij48c3BhbiBzdHlsZT0iZm9udC1mYW1p bHk6JnF1b3Q7QXJpYWwmcXVvdDssJnF1b3Q7c2Fucy1zZXJpZiZxdW90Oztjb2xvcjojMjIyMjIy Ij5IaSB0aGVyZSwNCjxvOnA+PC9vOnA+PC9zcGFuPjwvcD4NCjwvdGQ+DQo8L3RyPg0KPC90Ym9k eT4NCjwvdGFibGU+DQo8L3RkPg0KPHRkIHN0eWxlPSJwYWRkaW5nOjBpbiAwaW4gMGluIDBpbiI+ PC90ZD4NCjx0ZCBzdHlsZT0icGFkZGluZzowaW4gMGluIDBpbiAwaW4iPjwvdGQ+DQo8dGQgc3R5 bGU9InBhZGRpbmc6MGluIDBpbiAwaW4gMGluIj48L3RkPg0KPC90cj4NCjx0cj4NCjx0ZCB2YWxp Z249InRvcCIgc3R5bGU9ImJhY2tncm91bmQ6I0Y1RjlGRDtwYWRkaW5nOjBpbiAwaW4gMGluIDBp biI+DQo8dGFibGUgY2xhc3M9Ik1zb05vcm1hbFRhYmxlIiBib3JkZXI9IjAiIGNlbGxzcGFjaW5n PSIwIiBjZWxscGFkZGluZz0iMCIgd2lkdGg9IjEwMCUiIHN0eWxlPSJ3aWR0aDoxMDAuMCUiPg0K PHRib2R5Pg0KPHRyPg0KPHRkIHZhbGlnbj0idG9wIiBzdHlsZT0icGFkZGluZzowaW4gMTEuMjVw dCAwaW4gMTEuMjVwdCI+DQo8cCBjbGFzcz0iTXNvTm9ybWFsIiBzdHlsZT0ibGluZS1oZWlnaHQ6 MTMuNXB0Ij48c3BhbiBzdHlsZT0iZm9udC1mYW1pbHk6JnF1b3Q7QXJpYWwmcXVvdDssJnF1b3Q7 c2Fucy1zZXJpZiZxdW90Oztjb2xvcjojNTc1NzU3Ij4mbmJzcDsNCjxvOnA+PC9vOnA+PC9zcGFu PjwvcD4NCjwvdGQ+DQo8L3RyPg0KPC90Ym9keT4NCjwvdGFibGU+DQo8L3RkPg0KPHRkIHN0eWxl PSJwYWRkaW5nOjBpbiAwaW4gMGluIDBpbiI+PC90ZD4NCjx0ZCBzdHlsZT0icGFkZGluZzowaW4g MGluIDBpbiAwaW4iPjwvdGQ+DQo8dGQgc3R5bGU9InBhZGRpbmc6MGluIDBpbiAwaW4gMGluIj48 L3RkPg0KPC90cj4NCjx0cj4NCjx0ZCB2YWxpZ249InRvcCIgc3R5bGU9ImJhY2tncm91bmQ6I0Y1 RjlGRDtwYWRkaW5nOjBpbiAxMS4yNXB0IDE1LjBwdCAxMS4yNXB0Ij4NCjx0YWJsZSBjbGFzcz0i TXNvTm9ybWFsVGFibGUiIGJvcmRlcj0iMCIgY2VsbHNwYWNpbmc9IjAiIGNlbGxwYWRkaW5nPSIw IiB3aWR0aD0iMTAwJSIgc3R5bGU9IndpZHRoOjEwMC4wJSI+DQo8dGJvZHk+DQo8dHI+DQo8dGQg dmFsaWduPSJ0b3AiIHN0eWxlPSJwYWRkaW5nOjBpbiAwaW4gMGluIDBpbiI+DQo8cCBjbGFzcz0i TXNvTm9ybWFsIiBzdHlsZT0ibGluZS1oZWlnaHQ6MTguNzVwdCI+PHNwYW4gc3R5bGU9ImZvbnQt ZmFtaWx5OiZxdW90O0FyaWFsJnF1b3Q7LCZxdW90O3NhbnMtc2VyaWYmcXVvdDs7Y29sb3I6IzU3 NTc1NyI+RGFuIFJvbWFzY2FudSAoZHJvbWFzY2FAYXZheWEuY29tKSBpbnZpdGVzIHlvdSB0byBw YXJ0aWNpcGF0ZSBpbiB0aGUgRG9vZGxlIHBvbGwNCjwvc3Bhbj48c3BhbiBzdHlsZT0iZm9udC1m YW1pbHk6JnF1b3Q7QXJpYWwmcXVvdDssJnF1b3Q7c2Fucy1zZXJpZiZxdW90Oztjb2xvcjojMjIy MjIyIj4mcXVvdDtTQUNNIFdHIFZpcnR1YWwgSW50ZXJpbSBNZWV0aW5nJnF1b3Q7PC9zcGFuPjxz cGFuIHN0eWxlPSJmb250LWZhbWlseTomcXVvdDtBcmlhbCZxdW90OywmcXVvdDtzYW5zLXNlcmlm JnF1b3Q7O2NvbG9yOiM1NzU3NTciPi4NCjxvOnA+PC9vOnA+PC9zcGFuPjwvcD4NCjwvdGQ+DQo8 L3RyPg0KPC90Ym9keT4NCjwvdGFibGU+DQo8L3RkPg0KPHRkIHN0eWxlPSJwYWRkaW5nOjBpbiAw aW4gMGluIDBpbiI+PC90ZD4NCjx0ZCBzdHlsZT0icGFkZGluZzowaW4gMGluIDBpbiAwaW4iPjwv dGQ+DQo8dGQgc3R5bGU9InBhZGRpbmc6MGluIDBpbiAwaW4gMGluIj48L3RkPg0KPC90cj4NCjx0 cj4NCjx0ZCBzdHlsZT0iYmFja2dyb3VuZDojREZFQ0ZDO3BhZGRpbmc6MGluIDBpbiAwaW4gMGlu Ij4NCjx0YWJsZSBjbGFzcz0iTXNvTm9ybWFsVGFibGUiIGJvcmRlcj0iMCIgY2VsbHNwYWNpbmc9 IjAiIGNlbGxwYWRkaW5nPSIwIiB3aWR0aD0iMTAwJSIgc3R5bGU9IndpZHRoOjEwMC4wJSI+DQo8 dGJvZHk+DQo8dHIgc3R5bGU9ImhlaWdodDo3LjVwdCI+DQo8dGQgd2lkdGg9IjE1IiBjb2xzcGFu PSIzIiB2YWxpZ249InRvcCIgc3R5bGU9IndpZHRoOjExLjI1cHQ7cGFkZGluZzowaW4gMGluIDBp biAwaW47aGVpZ2h0OjcuNXB0Ij4NCjwvdGQ+DQo8L3RyPg0KPHRyPg0KPHRkIHN0eWxlPSJwYWRk aW5nOjBpbiAwaW4gMGluIDBpbjtib3gtc2hhZG93OiAwcHggMHB4IDJweCAwIHJnYigwLCAwLCAw LjI4KTtib3JkZXItcmFkaXVzOiAzcHgiPg0KPHRhYmxlIGNsYXNzPSJNc29Ob3JtYWxUYWJsZSIg Ym9yZGVyPSIwIiBjZWxscGFkZGluZz0iMCIgc3R5bGU9ImJvcmRlci1zcGFjaW5nOiAxNHB4IDBw eCI+DQo8dGJvZHk+DQo8dHI+DQo8dGQgc3R5bGU9ImJhY2tncm91bmQ6IzAwNjZERDtwYWRkaW5n OjMuMHB0IDUuMjVwdCAzLjBwdCA1LjI1cHQiPg0KPHAgY2xhc3M9Ik1zb05vcm1hbCIgc3R5bGU9 Im1zby1tYXJnaW4tdG9wLWFsdDowaW47bWFyZ2luLXJpZ2h0OjIuMjVwdDttYXJnaW4tYm90dG9t OjBpbjttYXJnaW4tbGVmdDoxMy41cHQ7bWFyZ2luLWJvdHRvbTouMDAwMXB0O2xpbmUtaGVpZ2h0 OjEzLjVwdCI+DQo8Yj48c3BhbiBzdHlsZT0iZm9udC1zaXplOjEwLjVwdDtmb250LWZhbWlseTom cXVvdDtBcmlhbCZxdW90OywmcXVvdDtzYW5zLXNlcmlmJnF1b3Q7Ij48YSBocmVmPSJodHRwczov L2Rvb2RsZS5jb20vdm53a3R0Nm1zZnB4YXZnej90bWFpbD1wb2xsX2ludml0ZWNvbnRhY3RfcGFy dGljaXBhbnRfaW52aXRhdGlvbiZhbXA7dGxpbms9cG9sbGJ0biI+PHNwYW4gc3R5bGU9ImNvbG9y OndoaXRlO3RleHQtZGVjb3JhdGlvbjpub25lIj5QYXJ0aWNpcGF0ZSZuYnNwO25vdzwvc3Bhbj48 L2E+DQo8bzpwPjwvbzpwPjwvc3Bhbj48L2I+PC9wPg0KPC90ZD4NCjwvdHI+DQo8L3Rib2R5Pg0K PC90YWJsZT4NCjwvdGQ+DQo8dGQgc3R5bGU9InBhZGRpbmc6MGluIDBpbiAwaW4gMGluIj48L3Rk Pg0KPHRkIHN0eWxlPSJwYWRkaW5nOjBpbiAwaW4gMGluIDBpbiI+PC90ZD4NCjwvdHI+DQo8dHIg c3R5bGU9ImhlaWdodDo3LjVwdCI+DQo8dGQgd2lkdGg9IjE1IiBjb2xzcGFuPSIzIiB2YWxpZ249 InRvcCIgc3R5bGU9IndpZHRoOjExLjI1cHQ7cGFkZGluZzowaW4gMGluIDBpbiAwaW47aGVpZ2h0 OjcuNXB0Ij4NCjwvdGQ+DQo8L3RyPg0KPC90Ym9keT4NCjwvdGFibGU+DQo8L3RkPg0KPHRkIHN0 eWxlPSJwYWRkaW5nOjBpbiAwaW4gMGluIDBpbiI+PC90ZD4NCjx0ZCBzdHlsZT0icGFkZGluZzow aW4gMGluIDBpbiAwaW4iPjwvdGQ+DQo8dGQgc3R5bGU9InBhZGRpbmc6MGluIDBpbiAwaW4gMGlu Ij48L3RkPg0KPC90cj4NCjx0cj4NCjx0ZCB2YWxpZ249InRvcCIgc3R5bGU9ImJvcmRlcjpzb2xp ZCAjREZFQ0ZDIDEuMHB0O3BhZGRpbmc6MTEuMjVwdCAxMS4yNXB0IDExLjI1cHQgMTEuMjVwdCI+ DQo8dGFibGUgY2xhc3M9Ik1zb05vcm1hbFRhYmxlIiBib3JkZXI9IjAiIGNlbGxzcGFjaW5nPSIw IiBjZWxscGFkZGluZz0iMCIgd2lkdGg9IjEwMCUiIHN0eWxlPSJ3aWR0aDoxMDAuMCUiPg0KPHRi b2R5Pg0KPHRyPg0KPHRkIHdpZHRoPSIzNSIgdmFsaWduPSJ0b3AiIHN0eWxlPSJ3aWR0aDoyNi4y NXB0O3BhZGRpbmc6MGluIDBpbiAwaW4gMGluIj4NCjxwIGNsYXNzPSJNc29Ob3JtYWwiPjxzcGFu IHN0eWxlPSJib3JkZXI6c29saWQgd2luZG93dGV4dCAxLjBwdDtwYWRkaW5nOjBpbiI+PGltZyBi b3JkZXI9IjAiIHdpZHRoPSIxMDAiIGhlaWdodD0iMTAwIiBpZD0iX3gwMDAwX2kxMDI2IiBzcmM9 ImNpZDp+V1JEMDAwLmpwZyIgYWx0PSJJbWFnZSByZW1vdmVkIGJ5IHNlbmRlci4iPjwvc3Bhbj48 bzpwPjwvbzpwPjwvcD4NCjwvdGQ+DQo8dGQgdmFsaWduPSJ0b3AiIHN0eWxlPSJwYWRkaW5nOjBp biAwaW4gMGluIDBpbiI+DQo8ZGl2Pg0KPHAgY2xhc3M9Ik1zb05vcm1hbCIgc3R5bGU9ImxpbmUt aGVpZ2h0OjE2LjVwdCI+PHNwYW4gc3R5bGU9ImZvbnQtc2l6ZTo5LjBwdDtmb250LWZhbWlseTom cXVvdDtBcmlhbCZxdW90OywmcXVvdDtzYW5zLXNlcmlmJnF1b3Q7O2NvbG9yOiMyMjIyMjIiPldo YXQgaXMgRG9vZGxlPzwvc3Bhbj48c3BhbiBzdHlsZT0iZm9udC1zaXplOjkuMHB0O2ZvbnQtZmFt aWx5OiZxdW90O0FyaWFsJnF1b3Q7LCZxdW90O3NhbnMtc2VyaWYmcXVvdDs7Y29sb3I6IzU3NTc1 NyI+IERvb2RsZSBpcyBhIHdlYiBzZXJ2aWNlIHRoYXQgaGVscHMNCiBEYW4gUm9tYXNjYW51IHRv IGZpbmQgYSBzdWl0YWJsZSBkYXRlIGZvciBtZWV0aW5nIHdpdGggYSBncm91cCBvZiBwZW9wbGUu IDxhIGhyZWY9Imh0dHBzOi8vZG9vZGxlLmNvbS9tYWluLmh0bWw/dGxpbms9Y2hlY2tPdXRMaW5r JmFtcDt0bWFpbD1wb2xsX2ludml0ZWNvbnRhY3RfcGFydGljaXBhbnRfaW52aXRhdGlvbiI+DQpM ZWFybiBtb3JlIGFib3V0IGhvdyBEb29kbGUgd29ya3MuPC9hPjxvOnA+PC9vOnA+PC9zcGFuPjwv cD4NCjwvZGl2Pg0KPC90ZD4NCjwvdHI+DQo8L3Rib2R5Pg0KPC90YWJsZT4NCjwvdGQ+DQo8dGQg c3R5bGU9InBhZGRpbmc6MGluIDBpbiAwaW4gMGluIj48L3RkPg0KPHRkIHN0eWxlPSJwYWRkaW5n OjBpbiAwaW4gMGluIDBpbiI+PC90ZD4NCjx0ZCBzdHlsZT0icGFkZGluZzowaW4gMGluIDBpbiAw aW4iPjwvdGQ+DQo8L3RyPg0KPHRyPg0KPHRkIHZhbGlnbj0idG9wIiBzdHlsZT0iYm9yZGVyOm5v bmU7Ym9yZGVyLXRvcDpzb2xpZCAjRjVGOUZEIDEuMHB0O3BhZGRpbmc6MGluIDBpbiAwaW4gMGlu Ij4NCjxkaXYgYWxpZ249ImNlbnRlciI+DQo8dGFibGUgY2xhc3M9Ik1zb05vcm1hbFRhYmxlIiBi b3JkZXI9IjAiIGNlbGxzcGFjaW5nPSIwIiBjZWxscGFkZGluZz0iMCI+DQo8dGJvZHk+DQo8dHIg c3R5bGU9ImhlaWdodDouMjVpbiI+DQo8dGQgc3R5bGU9InBhZGRpbmc6MGluIDBpbiAwaW4gMGlu O2hlaWdodDouMjVpbiI+PC90ZD4NCjwvdHI+DQo8dHI+DQo8dGQgdmFsaWduPSJ0b3AiIHN0eWxl PSJwYWRkaW5nOjBpbiAxMS4yNXB0IDYuNzVwdCAxMS4yNXB0Ij4NCjxwIGNsYXNzPSJNc29Ob3Jt YWwiIHN0eWxlPSJsaW5lLWhlaWdodDoxMi4wcHQiPjxzcGFuIHN0eWxlPSJmb250LXNpemU6OS4w cHQ7Zm9udC1mYW1pbHk6JnF1b3Q7QXJpYWwmcXVvdDssJnF1b3Q7c2Fucy1zZXJpZiZxdW90Oztj b2xvcjojOTk5OTk5Ij5Zb3UgaGF2ZSByZWNlaXZlZCB0aGlzIGUtbWFpbCBiZWNhdXNlICZxdW90 O0RhbiBSb21hc2NhbnUmcXVvdDsgaGFzIGludml0ZWQgeW91IHRvIHBhcnRpY2lwYXRlIGluIHRo ZSBEb29kbGUgcG9sbCAmcXVvdDtTQUNNIFdHIFZpcnR1YWwgSW50ZXJpbSBNZWV0aW5nLiZxdW90 Ow0KPG86cD48L286cD48L3NwYW4+PC9wPg0KPC90ZD4NCjwvdHI+DQo8dHIgc3R5bGU9ImhlaWdo dDo5LjBwdCI+DQo8dGQgc3R5bGU9InBhZGRpbmc6MGluIDBpbiAwaW4gMGluO2hlaWdodDo5LjBw dCI+PC90ZD4NCjwvdHI+DQo8L3Rib2R5Pg0KPC90YWJsZT4NCjwvZGl2Pg0KPC90ZD4NCjx0ZCBz dHlsZT0icGFkZGluZzowaW4gMGluIDBpbiAwaW4iPjwvdGQ+DQo8dGQgc3R5bGU9InBhZGRpbmc6 MGluIDBpbiAwaW4gMGluIj48L3RkPg0KPHRkIHN0eWxlPSJwYWRkaW5nOjBpbiAwaW4gMGluIDBp biI+PC90ZD4NCjwvdHI+DQo8dHI+DQo8dGQgdmFsaWduPSJ0b3AiIHN0eWxlPSJib3JkZXI6bm9u ZTtib3JkZXItdG9wOnNvbGlkICNEREREREQgMS4wcHQ7cGFkZGluZzowaW4gMGluIDBpbiAwaW4i Pg0KPHRhYmxlIGNsYXNzPSJNc29Ob3JtYWxUYWJsZSIgYm9yZGVyPSIwIiBjZWxsc3BhY2luZz0i MCIgY2VsbHBhZGRpbmc9IjAiIHdpZHRoPSIxMDAlIiBzdHlsZT0id2lkdGg6MTAwLjAlIj4NCjx0 Ym9keT4NCjx0cj4NCjx0ZCB2YWxpZ249InRvcCIgc3R5bGU9InBhZGRpbmc6OS4wcHQgMTEuMjVw dCAxNS4wcHQgMTEuMjVwdCI+DQo8cCBjbGFzcz0iTXNvTm9ybWFsIiBzdHlsZT0ibGluZS1oZWln aHQ6MTIuNzVwdCI+PHNwYW4gc3R5bGU9ImZvbnQtc2l6ZTo5LjBwdDtmb250LWZhbWlseTomcXVv dDtBcmlhbCZxdW90OywmcXVvdDtzYW5zLXNlcmlmJnF1b3Q7O2NvbG9yOiM5OTk5OTkiPkRvb2Rs ZSBBRywgV2VyZHN0cmFzc2UgMjEsIDgwMjEgWsO8cmljaA0KPG86cD48L286cD48L3NwYW4+PC9w Pg0KPC90ZD4NCjwvdHI+DQo8L3Rib2R5Pg0KPC90YWJsZT4NCjwvdGQ+DQo8dGQgc3R5bGU9InBh ZGRpbmc6MGluIDBpbiAwaW4gMGluIj48L3RkPg0KPHRkIHN0eWxlPSJwYWRkaW5nOjBpbiAwaW4g MGluIDBpbiI+PC90ZD4NCjx0ZCBzdHlsZT0icGFkZGluZzowaW4gMGluIDBpbiAwaW4iPjwvdGQ+ DQo8L3RyPg0KPC90Ym9keT4NCjwvdGFibGU+DQo8L2Rpdj4NCjwvdGQ+DQo8L3RyPg0KPC90Ym9k eT4NCjwvdGFibGU+DQo8L2Rpdj4NCjxwIGNsYXNzPSJNc29Ob3JtYWwiPjxvOnA+Jm5ic3A7PC9v OnA+PC9wPg0KPC9kaXY+DQo8L2Rpdj4NCjwvZGl2Pg0KPC9ib2R5Pg0KPC9odG1sPg0K --_000_9904FB1B0159DA42B0B887B7FA8119CA1292FAF4AZFFEXMB04globa_-- --_005_9904FB1B0159DA42B0B887B7FA8119CA1292FAF4AZFFEXMB04globa_ Content-Type: image/jpeg; name="~WRD000.jpg" Content-Description: ~WRD000.jpg Content-Disposition: inline; filename="~WRD000.jpg"; size=823; creation-date="Thu, 07 Nov 2013 01:06:32 GMT"; modification-date="Thu, 07 Nov 2013 01:06:32 GMT" Content-ID: <~WRD000.jpg> Content-Transfer-Encoding: base64 /9j/4AAQSkZJRgABAQEAYABgAAD/2wBDAAgGBgcGBQgHBwcJCQgKDBQNDAsLDBkSEw8UHRofHh0a HBwgJC4nICIsIxwcKDcpLDAxNDQ0Hyc5PTgyPC4zNDL/2wBDAQkJCQwLDBgNDRgyIRwhMjIyMjIy MjIyMjIyMjIyMjIyMjIyMjIyMjIyMjIyMjIyMjIyMjIyMjIyMjIyMjIyMjL/wAARCABkAGQDASIA AhEBAxEB/8QAHwAAAQUBAQEBAQEAAAAAAAAAAAECAwQFBgcICQoL/8QAtRAAAgEDAwIEAwUFBAQA AAF9AQIDAAQRBRIhMUEGE1FhByJxFDKBkaEII0KxwRVS0fAkM2JyggkKFhcYGRolJicoKSo0NTY3 ODk6Q0RFRkdISUpTVFVWV1hZWmNkZWZnaGlqc3R1dnd4eXqDhIWGh4iJipKTlJWWl5iZmqKjpKWm p6ipqrKztLW2t7i5usLDxMXGx8jJytLT1NXW19jZ2uHi4+Tl5ufo6erx8vP09fb3+Pn6/8QAHwEA AwEBAQEBAQEBAQAAAAAAAAECAwQFBgcICQoL/8QAtREAAgECBAQDBAcFBAQAAQJ3AAECAxEEBSEx BhJBUQdhcRMiMoEIFEKRobHBCSMzUvAVYnLRChYkNOEl8RcYGRomJygpKjU2Nzg5OkNERUZHSElK U1RVVldYWVpjZGVmZ2hpanN0dXZ3eHl6goOEhYaHiImKkpOUlZaXmJmaoqOkpaanqKmqsrO0tba3 uLm6wsPExcbHyMnK0tPU1dbX2Nna4uPk5ebn6Onq8vP09fb3+Pn6/9oADAMBAAIRAxEAPwD3+iii gAooooAKKKKACiiigAooooAKKKKACiiigAooooAKKKKACiiigAooooAKKKKACiiigAooooAKKKKA CiiigAooooAKKKKACiiigAooooAKKKKACiiigAooooAKKKKACiiigAooooAKKKKACiiigAooooAK KKKACiiigAooooAKKKKACiiigAooooAKKKKACiiigAooooAKKKKACiiigAooooAKKKKACiiigAoo ooAKKKKACiiigAooooAKKKKACiiigD//2Q== --_005_9904FB1B0159DA42B0B887B7FA8119CA1292FAF4AZFFEXMB04globa_ Content-Type: image/jpeg; name="image001.jpg" Content-Description: image001.jpg Content-Disposition: inline; filename="image001.jpg"; size=344; creation-date="Thu, 07 Nov 2013 01:07:42 GMT"; modification-date="Thu, 07 Nov 2013 01:07:42 GMT" Content-ID: Content-Transfer-Encoding: base64 /9j/4AAQSkZJRgABAQEAYABgAAD/2wBDAAoHBwgHBgoICAgLCgoLDhgQDg0NDh0VFhEYIx8lJCIf IiEmKzcvJik0KSEiMEExNDk7Pj4+JS5ESUM8SDc9Pjv/wAALCAAaABoBAREA/8QAHwAAAQUBAQEB AQEAAAAAAAAAAAECAwQFBgcICQoL/8QAtRAAAgEDAwIEAwUFBAQAAAF9AQIDAAQRBRIhMUEGE1Fh ByJxFDKBkaEII0KxwRVS0fAkM2JyggkKFhcYGRolJicoKSo0NTY3ODk6Q0RFRkdISUpTVFVWV1hZ WmNkZWZnaGlqc3R1dnd4eXqDhIWGh4iJipKTlJWWl5iZmqKjpKWmp6ipqrKztLW2t7i5usLDxMXG x8jJytLT1NXW19jZ2uHi4+Tl5ufo6erx8vP09fb3+Pn6/9oACAEBAAA/APZqKKKKKKKKKKKKKKK/ /9k= --_005_9904FB1B0159DA42B0B887B7FA8119CA1292FAF4AZFFEXMB04globa_-- From frode.sorensen@npt.no Wed Nov 6 23:16:23 2013 Return-Path: X-Original-To: lmap@ietfa.amsl.com Delivered-To: lmap@ietfa.amsl.com Received: from localhost (localhost [127.0.0.1]) by ietfa.amsl.com (Postfix) with ESMTP id 01C5311E8242 for ; Wed, 6 Nov 2013 23:16:23 -0800 (PST) X-Virus-Scanned: amavisd-new at amsl.com X-Spam-Flag: NO X-Spam-Score: -3.01 X-Spam-Level: X-Spam-Status: No, score=-3.01 tagged_above=-999 required=5 tests=[AWL=-0.712, BAYES_00=-2.599, MIME_8BIT_HEADER=0.3, UNPARSEABLE_RELAY=0.001] Received: from mail.ietf.org ([12.22.58.30]) by localhost (ietfa.amsl.com [127.0.0.1]) (amavisd-new, port 10024) with ESMTP id XLQ8BuzP+tQo for ; Wed, 6 Nov 2013 23:16:18 -0800 (PST) Received: from mail1.bemta5.messagelabs.com (mail1.bemta5.messagelabs.com [195.245.231.141]) by ietfa.amsl.com (Postfix) with ESMTP id 6088D11E821A for ; Wed, 6 Nov 2013 23:16:13 -0800 (PST) Received: from [85.158.139.211:53878] by server-5.bemta-5.messagelabs.com id C0/AC-13073-C3E3B725; Thu, 07 Nov 2013 07:16:12 +0000 X-Env-Sender: frode.sorensen@npt.no X-Msg-Ref: server-13.tower-206.messagelabs.com!1383808571!502352!1 X-Originating-IP: [213.225.64.154] X-StarScan-Received: X-StarScan-Version: 6.9.13; banners=-,-,- X-VirusChecked: Checked Received: (qmail 15451 invoked from network); 7 Nov 2013 07:16:12 -0000 Received: from unknown (HELO EXCAS.npta.no) (213.225.64.154) by server-13.tower-206.messagelabs.com with AES128-SHA encrypted SMTP; 7 Nov 2013 07:16:12 -0000 Received: from EXMBX01.npta.no ([10.10.2.97]) by EXCAS.npta.no ([fe80::b007:5474:dccd:c4dd%11]) with mapi id 14.02.0347.000; Thu, 7 Nov 2013 08:16:11 +0100 From: =?iso-8859-1?Q?S=F8rensen=2C_Frode?= To: Nieminen Klaus , James Miller , "MORTON JR. ALFRED, (AL)" Thread-Topic: [lmap] LMAP regulator use case Thread-Index: Ac7VfioGb4cSexizQzSZrQa31mFanAFFb7bPABe4HpAAANCwAAABqgaAACLtcfA= Date: Thu, 7 Nov 2013 07:16:09 +0000 Message-ID: <793D91975B99224DA9777562FF192808A274B577@exmbx01> References: <793D91975B99224DA9777562FF192808A27468CB@exmbx01> <2845723087023D4CB5114223779FA9C8ABF5C7D1@njfpsrvexg8.research.att.com> <6AF4522BD5AB86429CFD3948A9AB2F2F3A228999@loota.laru.local> In-Reply-To: <6AF4522BD5AB86429CFD3948A9AB2F2F3A228999@loota.laru.local> Accept-Language: nb-NO, en-US Content-Language: nb-NO X-MS-Has-Attach: X-MS-TNEF-Correlator: x-originating-ip: [10.10.8.40] Content-Type: text/plain; charset="iso-8859-1" Content-Transfer-Encoding: quoted-printable MIME-Version: 1.0 Cc: "philip.eardley@bt.com" , "lmap@ietf.org" Subject: Re: [lmap] LMAP regulator use case X-BeenThere: lmap@ietf.org X-Mailman-Version: 2.1.12 Precedence: list List-Id: Large Scale Measurement of Access network Performance List-Unsubscribe: , List-Archive: List-Post: List-Help: List-Subscribe: , X-List-Received-Date: Thu, 07 Nov 2013 07:16:23 -0000 Thanks for your comments, Yes, I agree that these use cases are complex and challenging. Although a c= omplete surveillance of such matters is a research topic, sample tests are = performed today based on exiting measurement tools. As far as I understood,= FCC investigated manipulation of BitTorrent traffic a while ago, I am not = sure to what extent they measured in that case, but this shows anyway the r= elevance of the use case. I believe regulators are in a good position to ex= plain what the regulator use case should look like, but of course interest = and comments from other stakeholders are appreciated and welcome. Best, Frode -----Opprinnelig melding----- Fra: Nieminen Klaus [mailto:Klaus.Nieminen@ficora.fi]=20 Sendt: 6. november 2013 16:30 Til: James Miller; MORTON JR. ALFRED, (AL) Kopi: philip.eardley@bt.com; lmap@ietf.org; S=F8rensen, Frode Emne: RE: [lmap] LMAP regulator use case Dear all, I would also like to add that as far as I have understood, LMAP WG don't st= andardize performance metrics. Therefore I think that we don't have to worr= y that much about how hard it is to measure degradation of the Internet acc= ess or specific applications. In addition as James and Frode mentioned ther= e are already some tools available for this purpose.=20 I fully support James' last remark that "it is an evolving space but someth= ing relevant for regulators use case description." - Klaus -----Original Message----- From: James Miller [mailto:jamesmilleresquire@gmail.com]=20 Sent: 6. marraskuuta 2013 16:43 To: MORTON JR. ALFRED, (AL) Cc: philip.eardley@bt.com; lmap@ietf.org; frode.sorensen@npt.no Subject: Re: [lmap] LMAP regulator use case I agree with Frode's comments on the 2.2 points. I would add the quality o= f the comparability of data is crucial for a regulators use and extends to = technical neutrality across the access link technologies as well, eg abilit= y to compare docsis, DSL, fiber, fixed wireless or other solutions. Most s= tudies I am familiar with include present and post processing techniques to= support comparisons across major broadband technologies that include out o= f scope topics. With 4.4 or 4.5 I agree there are a mix of views but would suggest there ar= e production quality tools as well as more research level quality topics. = I believe many are likely to extend to discussion of ippm metrics. In addi= tion to the tools mentioned, nick weaver's netalizer suite is an example th= at tests for easy things (like port blocking) and harder more subtle effect= s (like throttling, caching and proxy effects), and would also mention work= by bitag and other groups trying to reach technical consensus on descripti= on of many traffic management practices. It is an evolving space but somet= hing relevant for regulators use case description. On Nov 6, 2013 8:36 AM, "MORTON, ALFRED C (AL)" wrote: Hi Frode, =09 I read your mail and particularly agree with Phil's comments excerpted bel= ow: =09 > 4.4 Detecting degradation of the Internet access > 4.5 Detecting degradation of specific applications > > I don't like these sections about testing for whether the ISP is being n= et > neutral. > I think it would be very hard to design tests. > In part this is because net neutrality is more of a political concept th= an > technical one (in my view) > > I tihnk some of the stuff in S4.4 is really part of 4.3, as it's a marke= t > development issue (the internet has got slower, and doesnt meet the poli= cy > objective of Superfast broadband by Year 2015 - or whatever)- =09 If one form of the absence of neutrality is censorship, even censorship is hard to define and can include performance degradation according to Nick Feamster's presentation at the IRTF open meeting: http://www.ietf.org/proceedings/88/slides/slides-88-irtfopen-3.pptx =09 I think it's fair to say this sort of detection testing is still a researc= h topic, Al =09 > -----Original Message----- > From: lmap-bounces@ietf.org [mailto:lmap-bounces@ietf.org] On Behalf Of > philip.eardley@bt.com > Sent: Tuesday, November 05, 2013 9:42 PM > To: frode.sorensen@npt.no; lmap@ietf.org > Subject: Re: [lmap] LMAP regulator use case > > Frode > thanks very much for the comments - and it is excellent to have detailed > text suggestions. > > 2.2 > I think i'm basically OK with this replacing the current S2.2 (including > the subsections) > I suggest that it should mention comparability in the final para (as wel= l > as in the penultimate one) - personally I think this is the most importa= nt > characteristic from a regulator's point of view, so that it's fair to > compare measurements they make across the different ISPs. > the one thing in the current S2.2 (actually in S2.2.3) that I think is > worth keeping is a mention of fixed and mobile. I think this could be > handled by adding to the Intro (S1) something like: "Measurements will > take place on both fixed and mobile access" > > > Happy to add a new chapter 4, but comments about some of your sub use > cases > > 4.1 Promoting competition through transparency > >>For competition to successfully discipline operators' behaviour in the > interests of their customers, end users must be fully aware of the > characteristics of the ISPs' access offers. > "must be fully aware" over-states things. > > >>The system can be operated by a regulator or a measurement provider. > add "on behalf of the regulator" > > >> However, the quality of an Internet access service is also determine= d > by the connectivity to the rest of the Internet. Therefore test > configurations which measure beyond the ISP's own network may also be > relevant in some cases. > true. perhaps worth mentioning that the ISPs can influence this to some > extent - how well they're connected to the Internet, how they cache > popular content etc. > > 4.2 Promoting end user empowerment > i think this can be deleted. this is the end user use case, not the > regulator use case. > > 4.3 Monitoring overall market development > good for me > > 4.4 Detecting degradation of the Internet access > 4.5 Detecting degradation of specific applications > > I don't like these sections about testing for whether the ISP is being n= et > neutral. > I think it would be very hard to design tests. > In part this is because net neutrality is more of a political concept th= an > technical one (in my view) > > i tihnk some of the stuff in S4.4 is really part of 4.3, as it's a marke= t > development issue (the internet has got slower, and doesnt meet the poli= cy > objective of Superfast broadband by Year 2015 - or whatever)- > > best wishes > phil > > ________________________________________ > From: lmap-bounces@ietf.org [lmap-bounces@ietf.org] On Behalf Of S=F8ren= sen, > Frode [frode.sorensen@npt.no] > Sent: 30 October 2013 14:55 > To: lmap@ietf.org > Subject: [lmap] LMAP regulator use case > > Marc, Phil, Trevor, all, > > I have prepared a proposed description of a regulator use case for the > LMAP use case ID. I noticed the "advertisement" for further input from > regulators for the use case a couple of weeks ago. Finally I have manage= d > to find sufficient time to contribute to the ID, supported by a couple o= f > "regulator friends" from the Net Neutrality Working Group of the BEREC > organization, representing European national regulators. The proposal > consists of two parts; a modification of section 2.2 "Regulators", plus = a > new chapter after chapter 3 "Details of ISP use case", titled "Details o= f > regulator use case". > > Thanks, > Frode > > > Proposed update: > > 2.2 Regulators > ------------------- > Regulators in jurisdictions around the world are responding to consumers= ' > adoption of Internet access services for traditional telecommunications > and media services by promoting competition among providers of electroni= c > communications, to ensure that users derive maximum benefit in terms of > choice, price, and quality. > > Some jurisdictions have responded to a need for greater information abou= t > Internet access service performance in the development of regulatory > policies and approaches for broadband technologies by developing large- > scale measurement programs. Programs such as the U.S. Federal > Communications Commission's Measuring Broadband America, European > Commission's Quality of Broadband Services in the EU reports and a growi= ng > list of other programs employ a diverse set of operational and technical > approaches to gathering data to perform analysis and reporting on divers= e > aspects of broadband performance. > > While each jurisdiction responds to distinct consumer, industry, and > regulatory concerns, much commonality exists in the need to produce > datasets that are able to compare multiple Internet access service > providers, diverse technical solutions, geographic and regional > distributions, and marketed and provisioned levels and combinations of > broadband Internet access services. In some jurisdictions, the role of > measuring is provided by a measurement provider. > > Measurement providers measure network performance from users towards > multiple content and application providers, included dedicated test > measurement servers, to show a performance of the actual Internet access > service provided by different ISPs. Users need to know the performance > that are achieving from their own ISP. In addition, they need to know th= e > performance of other ISPs of same location as background information for > selecting their ISP. Measurement providers will provide measurement > results with associated measurement methods and measurement metrics. > > From a consumer perspective, the differentiation between fixed and mobil= e > (cellular) Internet access services is blurring as the applications used > are very similar. Hence, similar measurements will take place on both > fixed and mobile Internet access services. > > Regulators role in the development and enforcement of broadband Internet > access service policies also require that the measurement approaches mee= t > a high level of verifiability, accuracy and provider-independence to > support valid and meaningful comparisons of Internet access service > performance > > LMAP standards could answer regulators shared needs by providing scalabl= e, > cost-effective, scientifically robust solutions to the measurement and > collection of broadband Internet access service performance information. > > > Proposed new chapter: > > 4 Details of regulator use case > --------------------------------------- > 4.1 Promoting competition through transparency > --------------------------------------------------------------- > Competition plays a vital role in regulation of the electronic > communications markets. For competition to successfully discipline > operators' behaviour in the interests of their customers, end users must > be fully aware of the characteristics of the ISPs' access offers. In som= e > jurisdictions regulators mandate transparent information made available > about service offers. > > End users need effective transparency to be able to make informed choice= s > throughout the different stages of their relationship with ISPs, when > selecting Internet access service offers, and when considering switching > service offer within an ISP or to an alternative ISP. Quality informatio= n > about service offers can be provided based on performance measurements o= f > the services provided in the market. > > A set of measurement parameters and associated measurement methods are > used over time, e.g. speed, delay, and jitter. Then the measurement raw > data are collected and go through statistical post-processing before the > results can be published in an Internet access service quality index to > facilitate end users' choice of service provider and offer. > > In order to make information on offerings more meaningful and comparable > to end users, common frames of reference on Internet access service > quality metrics are essential. An example would be that in relation to > access speed, the actual download and upload speeds are measured and > published, not only the maximum speed. > > A measurement system that monitor Internet access services and collect > quality information can typically consist of a number of measurement > probes and one or more test servers located at peering points. The syste= m > can be operated by a regulator or a measurement provider. Number and > distribution of probes follows specific requirements depending on the > scope and the desired statistical reliability of the measurement campaig= n. > > In many test configurations, the measurement results will only reflect t= he > performance of ISP's own network. However, the quality of an Internet > access service is also determined by the connectivity to the rest of the > Internet. Therefore test configurations which measure beyond the ISP's o= wn > network may also be relevant in some cases. > > > 4.2 Promoting end user empowerment > -------------------------------------------------- > In addition to the advantages that end users receive from the competitio= n > in the market, through availability of a plethora of ISPs and Internet > access service offers, facilitated by transparent quality information > about these offers at a general level, end users would further leverage = on > information about specific services they subscribe to, or which they are > considering to subscribe to. > > Performance of Internet access service offers vary based on where the > service is provided (geographical and topological aspects), how it is > provided (e.g. access technology and ISP's interconnection to the > Internet) and when it is provided (variation through the day and over th= e > week). Therefore there is a need to obtain more detailed quality > information about specific service offers. > > Quality measurement tools can be made available for end users to monitor > their Internet access service, giving information about the speed and > other quality metrics of the subscribed service. Speed meters and simila= r > tools can be provided by regulators or other third parties, but also > directly by the ISP for the benefit of the subscriber. > > Such measurement tools assist end users in verifying the performance of > the Internet access service provided, comparing the measurement results > achieved with the information given in the contract. If the contracted > service quality level is not met, end users can use these results when > complaining to the ISP to achieve improvement and compensation. > > It may also help end users to assess whether they have chosen the packag= e > which best suits their specific needs. With their performance results at > hand, they can benefit from quality evaluation guides, which offer > comparisons based on their specific usage profiles. This may enable them > to check whether their current subscription is the most appropriate, and > assist them to switch package or provider if it is in their best interes= t. > > When provided by regulators and other third parties, such quality > measurement tools can also be used to compare performance between ISPs. = It > should, though, be recognised that measurement results can be affected b= y > factors outside the ISP's control, related for instance with local netwo= rk > connection (e.g. Wi-Fi), equipment and software being used. > > > 4.3 Monitoring overall market development > --------------------------------------------------------- > Governments sometimes set strategic goals for high-speed Internet access > penetration as an important component of the economic, cultural and soci= al > development of the society. To evaluate the effect of the stimulated > growth over time, broadband Internet access take-up and penetration of > high-speed access can be monitored through measurement campaigns. > > An example of such an initiative is the "Digital Agenda for Europe" whic= h > was adopted in 2010, to achieve universal broadband access. The goal is = to > achieve by 2020, access for all Europeans to Internet access speeds of 3= 0 > Mbps or above, and 50% or more of European households subscribing to > Internet connections above 100 Mbps. > > To monitor actual broadband Internet access performance in a specific > country or a region, extensive measurement campaigns are needed. A panel > can be built based on operators and packages in the market, spread over > urban, suburban and rural areas. Probes can then be distributed to the > participants of the campaign. > > Periodic tests running on the probes can for example measure actual spee= d > at peak and off-peak hours, but also other detailed quality metrics like > delay and jitter. Collected data goes afterwards through statistical > analysis, deriving estimates for the whole population which can then be > presented and published regularly. > > Using a harmonized or standardised measurement methodology, or even a > common quality measurement platform, measurement results could also be > used for benchmarking of providers and/or countries. > > > 4.4 Detecting degradation of the Internet access > -------------------------------------------------------------- > Regulation related to net neutrality and the open Internet has been > introduced in some jurisdictions' Internet policy, and monitoring method= s > for detection of potential net neutrality breaches is needed to follow u= p > such regulations with concrete actions. > > > Net neutrality is typically related to equal treatment of traffic > transmitted over the Internet access service, while other traffic > transmitted in parallel over the end user's broadband connection e.g. to > provide IPTV in IP networks separated from the Internet (also referred t= o > as "closed IP networks") can be exempted from net neutrality regulation. > Services provided in parallel to the Internet access services are referr= ed > to as specialised services (ref. BEREC and FCC). > > In order for such regulatory service architecture to protect net > neutrality and the open Internet, it is essential that the specialised > services are not provided at the expense of the Internet access service. > Therefore regulators may need to measure performance of the Internet > access service over time, and thereby detect whether this service is > becoming degraded. > > When regulators monitor this development to ensure net neutrality and th= e > open Internet, they can check e.g. whether Internet access service > performance and level of quality reflect advances in technology and that > this is not impaired by specialised services. Comparison between ISPs or > between different countries may also be relevant for this kind of > evaluation. > > Based on such Internet access service quality monitoring, regulators wil= l > typically report results and findings regularly, e.g. on an annual basis= . > In some jurisdictions regulators may also eventually impose minimum > quality of service requirements on ISPs in order to prevent degradation = of > Internet access services. > > > 4.5 Detecting degradation of specific applications > --------------------------------------------------------------- > In the regulatory context related to net neutrality and the open Interne= t, > in addition to the monitoring to detect potential degradation of the > access service as a whole described above, it is essential to perform > monitoring to detect potential degradation of individual applications > using the Internet access. > > Since net neutrality relates to equal treatment of traffic transmitted > over the Internet access service, the treatment of different application= s' > traffic is in the core of net neutrality. When traffic originating from > different applications are treated independent of which application each > IP packet belongs to, this is referred to as application agnosticism. > > Examples of departure from application agnosticism are blocking or > throttling of traffic from specific applications, but also preferential > treatment of specific applications. If traffic is forwarded at different > priority levels, traffic having higher priority level implies that other > traffic has lower priority. Detection of such traffic management practic= es > does not necessarily imply breaches to net neutrality, but these > observations can be used as an input to the regulatory evaluation of the > practice. > > When regulators monitor traffic management practices to ensure net > neutrality and the open Internet, they can check compliance with criteri= a > defining what is considered reasonable traffic management. Measurement > systems designed to monitor this kind of effects need to send applicatio= n- > specific traffic and then measure in detail the behaviour of the differe= nt > packets receive when transferred over the Internet access service. > > Today there exist measurement tools that can detect differentiated > treatment of individual applications. Another method that can be used is > to measure performance at application layer, e.g. assisted by content an= d > application providers, but also passive measurements are foreseen as a > promising approach for monitoring of application-specific treatment. > > _______________________________________________ > lmap mailing list > lmap@ietf.org > https://www.ietf.org/mailman/listinfo/lmap > _______________________________________________ > lmap mailing list > lmap@ietf.org > https://www.ietf.org/mailman/listinfo/lmap _______________________________________________ lmap mailing list lmap@ietf.org https://www.ietf.org/mailman/listinfo/lmap =09 From jamesmilleresquire@gmail.com Thu Nov 7 07:03:03 2013 Return-Path: X-Original-To: lmap@ietfa.amsl.com Delivered-To: lmap@ietfa.amsl.com Received: from localhost (localhost [127.0.0.1]) by ietfa.amsl.com (Postfix) with ESMTP id 0335221E823B for ; Thu, 7 Nov 2013 07:03:03 -0800 (PST) X-Virus-Scanned: amavisd-new at amsl.com X-Spam-Flag: NO X-Spam-Score: -2.449 X-Spam-Level: X-Spam-Status: No, score=-2.449 tagged_above=-999 required=5 tests=[AWL=-0.150, BAYES_00=-2.599, HTML_MESSAGE=0.001, MIME_8BIT_HEADER=0.3, NO_RELAYS=-0.001] Received: from mail.ietf.org ([12.22.58.30]) by localhost (ietfa.amsl.com [127.0.0.1]) (amavisd-new, port 10024) with ESMTP id 8b55PYGBdX-0 for ; Thu, 7 Nov 2013 07:03:00 -0800 (PST) Received: from mail-wg0-x235.google.com (mail-wg0-x235.google.com [IPv6:2a00:1450:400c:c00::235]) by ietfa.amsl.com (Postfix) with ESMTP id 4DF1621E823F for ; Thu, 7 Nov 2013 07:02:19 -0800 (PST) Received: by mail-wg0-f53.google.com with SMTP id y10so647115wgg.20 for ; Thu, 07 Nov 2013 07:02:16 -0800 (PST) DKIM-Signature: v=1; a=rsa-sha256; c=relaxed/relaxed; d=gmail.com; s=20120113; h=mime-version:in-reply-to:references:from:date:message-id:subject:to :cc:content-type; bh=jejVcH9OhVAVeW5UK0I2ai9h2ESHkvri05s8ccy5HXU=; b=FJHiunqsKQj5I/FoJRM9whAL2fHZ6Xwj4twQ+KNnyFb7AT9CppqSWS9qH/WX2WDChG BhIZ65v9OsDT2GVAhd+VIspFxxyEl+th2kBmtOlVGnEBL802EFLSnCNixDRqXmMVwN0B SA7639Z3Te+g/82oRTuco5osukcH7jvyiCwXAsnMVcJUOhsClWXP2m+qHZNA5RKOH8kS Ma8vVBeVT2HPRbt1iVPm72mpjX6C8k8vSdI/qzmnf4ICP7+AS/eSnu1diXwmb7Ja7/b1 4jEWhW+GFaFFpDAnCdXTFShzelT0O5Ggn0IMiBA6Ur2gH5fDRluMuUFiuIqktADrQwAn HAvg== X-Received: by 10.180.12.14 with SMTP id u14mr3029161wib.63.1383836536771; Thu, 07 Nov 2013 07:02:16 -0800 (PST) MIME-Version: 1.0 Received: by 10.180.211.7 with HTTP; Thu, 7 Nov 2013 07:01:36 -0800 (PST) In-Reply-To: <793D91975B99224DA9777562FF192808A274B577@exmbx01> References: <793D91975B99224DA9777562FF192808A27468CB@exmbx01> <2845723087023D4CB5114223779FA9C8ABF5C7D1@njfpsrvexg8.research.att.com> <6AF4522BD5AB86429CFD3948A9AB2F2F3A228999@loota.laru.local> <793D91975B99224DA9777562FF192808A274B577@exmbx01> From: James Miller Date: Thu, 7 Nov 2013 10:01:36 -0500 Message-ID: To: =?ISO-8859-1?Q?S=F8rensen=2C_Frode?= Content-Type: multipart/alternative; boundary=001a11c23ed2339faa04ea978dac Cc: Nieminen Klaus , "MORTON JR. ALFRED, \(AL\)" , "philip.eardley@bt.com" , "lmap@ietf.org" Subject: Re: [lmap] LMAP regulator use case X-BeenThere: lmap@ietf.org X-Mailman-Version: 2.1.12 Precedence: list List-Id: Large Scale Measurement of Access network Performance List-Unsubscribe: , List-Archive: List-Post: List-Help: List-Subscribe: , X-List-Received-Date: Thu, 07 Nov 2013 15:03:03 -0000 --001a11c23ed2339faa04ea978dac Content-Type: text/plain; charset=ISO-8859-1 Content-Transfer-Encoding: quoted-printable I am very much overloaded right now with.. some news to follow but would be happy to do a webex or email chain discussing the "Special Studies" we have been contemplating on this point. -- JamesMillerEsquire@Gmail.com "Wovon man nicht sprechen kann, dar=FCber mu=DF man schweigen." (What we cannot speak about we must pass over in silence. ) Ludwig Josef Johann Wittgenstein On Thu, Nov 7, 2013 at 2:16 AM, S=F8rensen, Frode wr= ote: > Thanks for your comments, > > Yes, I agree that these use cases are complex and challenging. Although a > complete surveillance of such matters is a research topic, sample tests a= re > performed today based on exiting measurement tools. As far as I understoo= d, > FCC investigated manipulation of BitTorrent traffic a while ago, I am not > sure to what extent they measured in that case, but this shows anyway the > relevance of the use case. I believe regulators are in a good position to > explain what the regulator use case should look like, but of course > interest and comments from other stakeholders are appreciated and welcome= . > > Best, > Frode > > > -----Opprinnelig melding----- > Fra: Nieminen Klaus [mailto:Klaus.Nieminen@ficora.fi] > Sendt: 6. november 2013 16:30 > Til: James Miller; MORTON JR. ALFRED, (AL) > Kopi: philip.eardley@bt.com; lmap@ietf.org; S=F8rensen, Frode > Emne: RE: [lmap] LMAP regulator use case > > Dear all, > > I would also like to add that as far as I have understood, LMAP WG don't > standardize performance metrics. Therefore I think that we don't have to > worry that much about how hard it is to measure degradation of the Intern= et > access or specific applications. In addition as James and Frode mentioned > there are already some tools available for this purpose. > > I fully support James' last remark that "it is an evolving space but > something relevant for regulators use case description." > > - Klaus > > -----Original Message----- > From: James Miller [mailto:jamesmilleresquire@gmail.com] > Sent: 6. marraskuuta 2013 16:43 > To: MORTON JR. ALFRED, (AL) > Cc: philip.eardley@bt.com; lmap@ietf.org; frode.sorensen@npt.no > Subject: Re: [lmap] LMAP regulator use case > > I agree with Frode's comments on the 2.2 points. I would add the quality > of the comparability of data is crucial for a regulators use and extends = to > technical neutrality across the access link technologies as well, eg > ability to compare docsis, DSL, fiber, fixed wireless or other solutions. > Most studies I am familiar with include present and post processing > techniques to support comparisons across major broadband technologies tha= t > include out of scope topics. > > With 4.4 or 4.5 I agree there are a mix of views but would suggest there > are production quality tools as well as more research level quality topic= s. > I believe many are likely to extend to discussion of ippm metrics. In > addition to the tools mentioned, nick weaver's netalizer suite is an > example that tests for easy things (like port blocking) and harder more > subtle effects (like throttling, caching and proxy effects), and would al= so > mention work by bitag and other groups trying to reach technical consensu= s > on description of many traffic management practices. It is an evolving > space but something relevant for regulators use case description. > > > On Nov 6, 2013 8:36 AM, "MORTON, ALFRED C (AL)" wrote: > > > Hi Frode, > > I read your mail and particularly agree with Phil's comments > excerpted below: > > > 4.4 Detecting degradation of the Internet access > > 4.5 Detecting degradation of specific applications > > > > I don't like these sections about testing for whether the ISP i= s > being net > > neutral. > > I think it would be very hard to design tests. > > In part this is because net neutrality is more of a political > concept than > > technical one (in my view) > > > > I tihnk some of the stuff in S4.4 is really part of 4.3, as it'= s > a market > > development issue (the internet has got slower, and doesnt meet > the policy > > objective of Superfast broadband by Year 2015 - or whatever)- > > If one form of the absence of neutrality is censorship, even > censorship is hard to define and can include performance > degradation > according to Nick Feamster's presentation at the IRTF open meetin= g: > > http://www.ietf.org/proceedings/88/slides/slides-88-irtfopen-3.pptx > > I think it's fair to say this sort of detection testing is still = a > research topic, > Al > > > -----Original Message----- > > From: lmap-bounces@ietf.org [mailto:lmap-bounces@ietf.org] On > Behalf Of > > philip.eardley@bt.com > > Sent: Tuesday, November 05, 2013 9:42 PM > > To: frode.sorensen@npt.no; lmap@ietf.org > > Subject: Re: [lmap] LMAP regulator use case > > > > Frode > > thanks very much for the comments - and it is excellent to have > detailed > > text suggestions. > > > > 2.2 > > I think i'm basically OK with this replacing the current S2.2 > (including > > the subsections) > > I suggest that it should mention comparability in the final par= a > (as well > > as in the penultimate one) - personally I think this is the mos= t > important > > characteristic from a regulator's point of view, so that it's > fair to > > compare measurements they make across the different ISPs. > > the one thing in the current S2.2 (actually in S2.2.3) that I > think is > > worth keeping is a mention of fixed and mobile. I think this > could be > > handled by adding to the Intro (S1) something like: > "Measurements will > > take place on both fixed and mobile access" > > > > > > Happy to add a new chapter 4, but comments about some of your > sub use > > cases > > > > 4.1 Promoting competition through transparency > > >>For competition to successfully discipline operators' > behaviour in the > > interests of their customers, end users must be fully aware of > the > > characteristics of the ISPs' access offers. > > "must be fully aware" over-states things. > > > > >>The system can be operated by a regulator or a measurement > provider. > > add "on behalf of the regulator" > > > > >> However, the quality of an Internet access service is also > determined > > by the connectivity to the rest of the Internet. Therefore test > > configurations which measure beyond the ISP's own network may > also be > > relevant in some cases. > > true. perhaps worth mentioning that the ISPs can influence this > to some > > extent - how well they're connected to the Internet, how they > cache > > popular content etc. > > > > 4.2 Promoting end user empowerment > > i think this can be deleted. this is the end user use case, not > the > > regulator use case. > > > > 4.3 Monitoring overall market development > > good for me > > > > 4.4 Detecting degradation of the Internet access > > 4.5 Detecting degradation of specific applications > > > > I don't like these sections about testing for whether the ISP i= s > being net > > neutral. > > I think it would be very hard to design tests. > > In part this is because net neutrality is more of a political > concept than > > technical one (in my view) > > > > i tihnk some of the stuff in S4.4 is really part of 4.3, as it'= s > a market > > development issue (the internet has got slower, and doesnt meet > the policy > > objective of Superfast broadband by Year 2015 - or whatever)- > > > > best wishes > > phil > > > > ________________________________________ > > From: lmap-bounces@ietf.org [lmap-bounces@ietf.org] On Behalf > Of S=F8rensen, > > Frode [frode.sorensen@npt.no] > > Sent: 30 October 2013 14:55 > > To: lmap@ietf.org > > Subject: [lmap] LMAP regulator use case > > > > Marc, Phil, Trevor, all, > > > > I have prepared a proposed description of a regulator use case > for the > > LMAP use case ID. I noticed the "advertisement" for further > input from > > regulators for the use case a couple of weeks ago. Finally I > have managed > > to find sufficient time to contribute to the ID, supported by a > couple of > > "regulator friends" from the Net Neutrality Working Group of th= e > BEREC > > organization, representing European national regulators. The > proposal > > consists of two parts; a modification of section 2.2 > "Regulators", plus a > > new chapter after chapter 3 "Details of ISP use case", titled > "Details of > > regulator use case". > > > > Thanks, > > Frode > > > > > > Proposed update: > > > > 2.2 Regulators > > ------------------- > > Regulators in jurisdictions around the world are responding to > consumers' > > adoption of Internet access services for traditional > telecommunications > > and media services by promoting competition among providers of > electronic > > communications, to ensure that users derive maximum benefit in > terms of > > choice, price, and quality. > > > > Some jurisdictions have responded to a need for greater > information about > > Internet access service performance in the development of > regulatory > > policies and approaches for broadband technologies by developin= g > large- > > scale measurement programs. Programs such as the U.S. Federal > > Communications Commission's Measuring Broadband America, Europe= an > > Commission's Quality of Broadband Services in the EU reports an= d > a growing > > list of other programs employ a diverse set of operational and > technical > > approaches to gathering data to perform analysis and reporting > on diverse > > aspects of broadband performance. > > > > While each jurisdiction responds to distinct consumer, industry= , > and > > regulatory concerns, much commonality exists in the need to > produce > > datasets that are able to compare multiple Internet access > service > > providers, diverse technical solutions, geographic and regional > > distributions, and marketed and provisioned levels and > combinations of > > broadband Internet access services. In some jurisdictions, the > role of > > measuring is provided by a measurement provider. > > > > Measurement providers measure network performance from users > towards > > multiple content and application providers, included dedicated > test > > measurement servers, to show a performance of the actual > Internet access > > service provided by different ISPs. Users need to know the > performance > > that are achieving from their own ISP. In addition, they need t= o > know the > > performance of other ISPs of same location as background > information for > > selecting their ISP. Measurement providers will provide > measurement > > results with associated measurement methods and measurement > metrics. > > > > From a consumer perspective, the differentiation between fixed > and mobile > > (cellular) Internet access services is blurring as the > applications used > > are very similar. Hence, similar measurements will take place o= n > both > > fixed and mobile Internet access services. > > > > Regulators role in the development and enforcement of broadband > Internet > > access service policies also require that the measurement > approaches meet > > a high level of verifiability, accuracy and > provider-independence to > > support valid and meaningful comparisons of Internet access > service > > performance > > > > LMAP standards could answer regulators shared needs by providin= g > scalable, > > cost-effective, scientifically robust solutions to the > measurement and > > collection of broadband Internet access service performance > information. > > > > > > Proposed new chapter: > > > > 4 Details of regulator use case > > --------------------------------------- > > 4.1 Promoting competition through transparency > > --------------------------------------------------------------- > > Competition plays a vital role in regulation of the electronic > > communications markets. For competition to successfully > discipline > > operators' behaviour in the interests of their customers, end > users must > > be fully aware of the characteristics of the ISPs' access > offers. In some > > jurisdictions regulators mandate transparent information made > available > > about service offers. > > > > End users need effective transparency to be able to make > informed choices > > throughout the different stages of their relationship with ISPs= , > when > > selecting Internet access service offers, and when considering > switching > > service offer within an ISP or to an alternative ISP. Quality > information > > about service offers can be provided based on performance > measurements of > > the services provided in the market. > > > > A set of measurement parameters and associated measurement > methods are > > used over time, e.g. speed, delay, and jitter. Then the > measurement raw > > data are collected and go through statistical post-processing > before the > > results can be published in an Internet access service quality > index to > > facilitate end users' choice of service provider and offer. > > > > In order to make information on offerings more meaningful and > comparable > > to end users, common frames of reference on Internet access > service > > quality metrics are essential. An example would be that in > relation to > > access speed, the actual download and upload speeds are measure= d > and > > published, not only the maximum speed. > > > > A measurement system that monitor Internet access services and > collect > > quality information can typically consist of a number of > measurement > > probes and one or more test servers located at peering points. > The system > > can be operated by a regulator or a measurement provider. > Number and > > distribution of probes follows specific requirements depending > on the > > scope and the desired statistical reliability of the measuremen= t > campaign. > > > > In many test configurations, the measurement results will only > reflect the > > performance of ISP's own network. However, the quality of an > Internet > > access service is also determined by the connectivity to the > rest of the > > Internet. Therefore test configurations which measure beyond th= e > ISP's own > > network may also be relevant in some cases. > > > > > > 4.2 Promoting end user empowerment > > -------------------------------------------------- > > In addition to the advantages that end users receive from the > competition > > in the market, through availability of a plethora of ISPs and > Internet > > access service offers, facilitated by transparent quality > information > > about these offers at a general level, end users would further > leverage on > > information about specific services they subscribe to, or which > they are > > considering to subscribe to. > > > > Performance of Internet access service offers vary based on > where the > > service is provided (geographical and topological aspects), how > it is > > provided (e.g. access technology and ISP's interconnection to t= he > > Internet) and when it is provided (variation through the day an= d > over the > > week). Therefore there is a need to obtain more detailed qualit= y > > information about specific service offers. > > > > Quality measurement tools can be made available for end users t= o > monitor > > their Internet access service, giving information about the > speed and > > other quality metrics of the subscribed service. Speed meters > and similar > > tools can be provided by regulators or other third parties, but > also > > directly by the ISP for the benefit of the subscriber. > > > > Such measurement tools assist end users in verifying the > performance of > > the Internet access service provided, comparing the measurement > results > > achieved with the information given in the contract. If the > contracted > > service quality level is not met, end users can use these > results when > > complaining to the ISP to achieve improvement and compensation. > > > > It may also help end users to assess whether they have chosen > the package > > which best suits their specific needs. With their performance > results at > > hand, they can benefit from quality evaluation guides, which > offer > > comparisons based on their specific usage profiles. This may > enable them > > to check whether their current subscription is the most > appropriate, and > > assist them to switch package or provider if it is in their bes= t > interest. > > > > When provided by regulators and other third parties, such quali= ty > > measurement tools can also be used to compare performance > between ISPs. It > > should, though, be recognised that measurement results can be > affected by > > factors outside the ISP's control, related for instance with > local network > > connection (e.g. Wi-Fi), equipment and software being used. > > > > > > 4.3 Monitoring overall market development > > --------------------------------------------------------- > > Governments sometimes set strategic goals for high-speed > Internet access > > penetration as an important component of the economic, cultural > and social > > development of the society. To evaluate the effect of the > stimulated > > growth over time, broadband Internet access take-up and > penetration of > > high-speed access can be monitored through measurement campaign= s. > > > > An example of such an initiative is the "Digital Agenda for > Europe" which > > was adopted in 2010, to achieve universal broadband access. The > goal is to > > achieve by 2020, access for all Europeans to Internet access > speeds of 30 > > Mbps or above, and 50% or more of European households > subscribing to > > Internet connections above 100 Mbps. > > > > To monitor actual broadband Internet access performance in a > specific > > country or a region, extensive measurement campaigns are needed= . > A panel > > can be built based on operators and packages in the market, > spread over > > urban, suburban and rural areas. Probes can then be distributed > to the > > participants of the campaign. > > > > Periodic tests running on the probes can for example measure > actual speed > > at peak and off-peak hours, but also other detailed quality > metrics like > > delay and jitter. Collected data goes afterwards through > statistical > > analysis, deriving estimates for the whole population which can > then be > > presented and published regularly. > > > > Using a harmonized or standardised measurement methodology, or > even a > > common quality measurement platform, measurement results could > also be > > used for benchmarking of providers and/or countries. > > > > > > 4.4 Detecting degradation of the Internet access > > -------------------------------------------------------------- > > Regulation related to net neutrality and the open Internet has > been > > introduced in some jurisdictions' Internet policy, and > monitoring methods > > for detection of potential net neutrality breaches is needed to > follow up > > such regulations with concrete actions. > > > > > > Net neutrality is typically related to equal treatment of traff= ic > > transmitted over the Internet access service, while other traff= ic > > transmitted in parallel over the end user's broadband connectio= n > e.g. to > > provide IPTV in IP networks separated from the Internet (also > referred to > > as "closed IP networks") can be exempted from net neutrality > regulation. > > Services provided in parallel to the Internet access services > are referred > > to as specialised services (ref. BEREC and FCC). > > > > In order for such regulatory service architecture to protect ne= t > > neutrality and the open Internet, it is essential that the > specialised > > services are not provided at the expense of the Internet access > service. > > Therefore regulators may need to measure performance of the > Internet > > access service over time, and thereby detect whether this > service is > > becoming degraded. > > > > When regulators monitor this development to ensure net > neutrality and the > > open Internet, they can check e.g. whether Internet access > service > > performance and level of quality reflect advances in technology > and that > > this is not impaired by specialised services. Comparison betwee= n > ISPs or > > between different countries may also be relevant for this kind = of > > evaluation. > > > > Based on such Internet access service quality monitoring, > regulators will > > typically report results and findings regularly, e.g. on an > annual basis. > > In some jurisdictions regulators may also eventually impose > minimum > > quality of service requirements on ISPs in order to prevent > degradation of > > Internet access services. > > > > > > 4.5 Detecting degradation of specific applications > > --------------------------------------------------------------- > > In the regulatory context related to net neutrality and the ope= n > Internet, > > in addition to the monitoring to detect potential degradation o= f > the > > access service as a whole described above, it is essential to > perform > > monitoring to detect potential degradation of individual > applications > > using the Internet access. > > > > Since net neutrality relates to equal treatment of traffic > transmitted > > over the Internet access service, the treatment of different > applications' > > traffic is in the core of net neutrality. When traffic > originating from > > different applications are treated independent of which > application each > > IP packet belongs to, this is referred to as application > agnosticism. > > > > Examples of departure from application agnosticism are blocking > or > > throttling of traffic from specific applications, but also > preferential > > treatment of specific applications. If traffic is forwarded at > different > > priority levels, traffic having higher priority level implies > that other > > traffic has lower priority. Detection of such traffic managemen= t > practices > > does not necessarily imply breaches to net neutrality, but thes= e > > observations can be used as an input to the regulatory > evaluation of the > > practice. > > > > When regulators monitor traffic management practices to ensure > net > > neutrality and the open Internet, they can check compliance wit= h > criteria > > defining what is considered reasonable traffic management. > Measurement > > systems designed to monitor this kind of effects need to send > application- > > specific traffic and then measure in detail the behaviour of th= e > different > > packets receive when transferred over the Internet access > service. > > > > Today there exist measurement tools that can detect > differentiated > > treatment of individual applications. Another method that can b= e > used is > > to measure performance at application layer, e.g. assisted by > content and > > application providers, but also passive measurements are > foreseen as a > > promising approach for monitoring of application-specific > treatment. > > > > _______________________________________________ > > lmap mailing list > > lmap@ietf.org > > https://www.ietf.org/mailman/listinfo/lmap > > _______________________________________________ > > lmap mailing list > > lmap@ietf.org > > https://www.ietf.org/mailman/listinfo/lmap > _______________________________________________ > lmap mailing list > lmap@ietf.org > https://www.ietf.org/mailman/listinfo/lmap > > > --001a11c23ed2339faa04ea978dac Content-Type: text/html; charset=ISO-8859-1 Content-Transfer-Encoding: quoted-printable
I am very much overloaded right now with.. some news to fo= llow but would be happy to do a webex or email chain discussing the "S= pecial Studies" we have been contemplating on this point.



--
JamesMillerEsquire@Gmail.com

<= /div>
"Wovon man nicht sprechen kann, dar=FCber mu=DF man schweige= n."
(What we cannot speak about we must pass over in silence. =A0)
Ludwig Josef Johann Wittgenstein


On Thu, Nov 7, 2013 at 2:16 AM, S=F8rens= en, Frode <frode.sorensen@npt.no> wrote:
Thanks for your comments,

Yes, I agree that these use cases are complex and challenging. Although a c= omplete surveillance of such matters is a research topic, sample tests are = performed today based on exiting measurement tools. As far as I understood,= FCC investigated manipulation of BitTorrent traffic a while ago, I am not = sure to what extent they measured in that case, but this shows anyway the r= elevance of the use case. I believe regulators are in a good position to ex= plain what the regulator use case should look like, but of course interest = and comments from other stakeholders are appreciated and welcome.

Best,
Frode


-----Opprinnelig melding-----
Fra: Nieminen Klaus [mailto:Kla= us.Nieminen@ficora.fi]
Sendt: 6. november 2013 16:30
Til: James Miller; MORTON JR. ALFRED, (AL)
Kopi: philip.eardley@bt.com; <= a href=3D"mailto:lmap@ietf.org">lmap@ietf.org; S=F8rensen, Frode
Emne: RE: [lmap] LMAP regulator use case

Dear all,

I would also like to add that as far as I have understood, LMAP WG don'= t standardize performance metrics. Therefore I think that we don't have= to worry that much about how hard it is to measure degradation of the Inte= rnet access or specific applications. In addition as James and Frode mentio= ned there are already some tools available for this purpose.

I fully support James' last remark that "it is an evolving space b= ut something relevant for regulators use case description."

- Klaus

-----Original Message-----
From: James Miller [mailto:= jamesmilleresquire@gmail.com]
Sent: 6. marraskuuta 2013 16:43
To: MORTON JR. ALFRED, (AL)
Cc: philip.eardley@bt.com; lmap@ietf.org; frode.sorensen@npt.no
Subject: Re: [lmap] LMAP regulator use case

I agree with Frode's comments on the 2.2 points. =A0I would add the qua= lity of the comparability of data is crucial for a regulators use and exten= ds to technical neutrality across the access link technologies as well, eg = ability to compare docsis, DSL, fiber, fixed wireless or other solutions. = =A0Most studies I am familiar with include present and post processing tech= niques to support comparisons across major broadband technologies that incl= ude out of scope topics.

With 4.4 or 4.5 I agree there are a mix of views but would suggest there ar= e production quality tools as well as more research level quality topics. = =A0I believe many are likely to extend to discussion of ippm metrics. =A0In= addition to the tools mentioned, nick weaver's netalizer suite is an e= xample that tests for easy things (like port blocking) and harder more subt= le effects (like throttling, caching and proxy effects), and would also men= tion work by bitag and other groups trying to reach technical consensus on = description of many traffic management practices. =A0It is an evolving spac= e but something relevant for regulators use case description.


On Nov 6, 2013 8:36 AM, "MORTON, ALFRED C (AL)" <acmorton@att.com> wrote:


=A0 =A0 =A0 =A0 Hi Frode,

=A0 =A0 =A0 =A0 I read your mail and particularly agree with Phil's com= ments excerpted below:

=A0 =A0 =A0 =A0 > 4.4 Detecting degradation of the Internet access
=A0 =A0 =A0 =A0 > 4.5 Detecting degradation of specific applications
=A0 =A0 =A0 =A0 >
=A0 =A0 =A0 =A0 > I don't like these sections about testing for whet= her the ISP is being net
=A0 =A0 =A0 =A0 > neutral.
=A0 =A0 =A0 =A0 > I think it would be very hard to design tests.
=A0 =A0 =A0 =A0 > In part this is because net neutrality is more of a po= litical concept than
=A0 =A0 =A0 =A0 > technical one (in my view)
=A0 =A0 =A0 =A0 >
=A0 =A0 =A0 =A0 > I tihnk some of the stuff in S4.4 is really part of 4.= 3, as it's a market
=A0 =A0 =A0 =A0 > development issue (the internet has got slower, and do= esnt meet the policy
=A0 =A0 =A0 =A0 > objective of Superfast broadband by Year 2015 - or wha= tever)-

=A0 =A0 =A0 =A0 If one form of the absence of neutrality is censorship, eve= n
=A0 =A0 =A0 =A0 censorship is hard to define and can include performance de= gradation
=A0 =A0 =A0 =A0 according to Nick Feamster's presentation at the IRTF o= pen meeting:
=A0 =A0 =A0 =A0 http://www.ietf.org/proceedings/88/s= lides/slides-88-irtfopen-3.pptx

=A0 =A0 =A0 =A0 I think it's fair to say this sort of detection testing= is still a research topic,
=A0 =A0 =A0 =A0 Al

=A0 =A0 =A0 =A0 > -----Original Message-----
=A0 =A0 =A0 =A0 > From: lmap-bo= unces@ietf.org [mailto:lmap-bo= unces@ietf.org] On Behalf Of
=A0 =A0 =A0 =A0 > philip.eardle= y@bt.com
=A0 =A0 =A0 =A0 > Sent: Tuesday, November 05, 2013 9:42 PM
=A0 =A0 =A0 =A0 > To: frode.sor= ensen@npt.no; lmap@ietf.org
=A0 =A0 =A0 =A0 > Subject: Re: [lmap] LMAP regulator use case
=A0 =A0 =A0 =A0 >
=A0 =A0 =A0 =A0 > Frode
=A0 =A0 =A0 =A0 > thanks very much for the comments - and it is excellen= t to have detailed
=A0 =A0 =A0 =A0 > text suggestions.
=A0 =A0 =A0 =A0 >
=A0 =A0 =A0 =A0 > 2.2
=A0 =A0 =A0 =A0 > I think i'm basically OK with this replacing the c= urrent S2.2 (including
=A0 =A0 =A0 =A0 > the subsections)
=A0 =A0 =A0 =A0 > I suggest that it should mention comparability in the = final para (as well
=A0 =A0 =A0 =A0 > as in the penultimate one) - personally I think this i= s the most important
=A0 =A0 =A0 =A0 > characteristic from a regulator's point of view, s= o that it's fair to
=A0 =A0 =A0 =A0 > compare measurements they make across the different IS= Ps.
=A0 =A0 =A0 =A0 > the one thing in the current S2.2 (actually in S2.2.3)= that I think is
=A0 =A0 =A0 =A0 > worth keeping is a mention of fixed and mobile. I thin= k this could be
=A0 =A0 =A0 =A0 > handled by adding to the Intro (S1) something like: &q= uot;Measurements will
=A0 =A0 =A0 =A0 > take place on both fixed and mobile access"
=A0 =A0 =A0 =A0 >
=A0 =A0 =A0 =A0 >
=A0 =A0 =A0 =A0 > Happy to add a new chapter 4, but comments about some = of your sub use
=A0 =A0 =A0 =A0 > cases
=A0 =A0 =A0 =A0 >
=A0 =A0 =A0 =A0 > 4.1 Promoting competition through transparency
=A0 =A0 =A0 =A0 > >>For competition to successfully discipline ope= rators' behaviour in the
=A0 =A0 =A0 =A0 > interests of their customers, end users must be fully = aware of the
=A0 =A0 =A0 =A0 > characteristics of the ISPs' access offers.
=A0 =A0 =A0 =A0 > "must be fully aware" over-states things. =A0 =A0 =A0 =A0 >
=A0 =A0 =A0 =A0 > >>The system can be operated by a regulator or a= measurement provider.
=A0 =A0 =A0 =A0 > add "on behalf of the regulator"
=A0 =A0 =A0 =A0 >
=A0 =A0 =A0 =A0 > >> =A0However, the quality of an Internet access= service is also determined
=A0 =A0 =A0 =A0 > by the connectivity to the rest of the Internet. There= fore test
=A0 =A0 =A0 =A0 > configurations which measure beyond the ISP's own = network may also be
=A0 =A0 =A0 =A0 > relevant in some cases.
=A0 =A0 =A0 =A0 > true. perhaps worth mentioning that the ISPs can influ= ence this to some
=A0 =A0 =A0 =A0 > extent - how well they're connected to the Interne= t, how they cache
=A0 =A0 =A0 =A0 > popular content etc.
=A0 =A0 =A0 =A0 >
=A0 =A0 =A0 =A0 > 4.2 Promoting end user empowerment
=A0 =A0 =A0 =A0 > i think this can be deleted. this is the end user use = case, not the
=A0 =A0 =A0 =A0 > regulator use case.
=A0 =A0 =A0 =A0 >
=A0 =A0 =A0 =A0 > 4.3 Monitoring overall market development
=A0 =A0 =A0 =A0 > good for me
=A0 =A0 =A0 =A0 >
=A0 =A0 =A0 =A0 > 4.4 Detecting degradation of the Internet access
=A0 =A0 =A0 =A0 > 4.5 Detecting degradation of specific applications
=A0 =A0 =A0 =A0 >
=A0 =A0 =A0 =A0 > I don't like these sections about testing for whet= her the ISP is being net
=A0 =A0 =A0 =A0 > neutral.
=A0 =A0 =A0 =A0 > I think it would be very hard to design tests.
=A0 =A0 =A0 =A0 > In part this is because net neutrality is more of a po= litical concept than
=A0 =A0 =A0 =A0 > technical one (in my view)
=A0 =A0 =A0 =A0 >
=A0 =A0 =A0 =A0 > i tihnk some of the stuff in S4.4 is really part of 4.= 3, as it's a market
=A0 =A0 =A0 =A0 > development issue (the internet has got slower, and do= esnt meet the policy
=A0 =A0 =A0 =A0 > objective of Superfast broadband by Year 2015 - or wha= tever)-
=A0 =A0 =A0 =A0 >
=A0 =A0 =A0 =A0 > best wishes
=A0 =A0 =A0 =A0 > phil
=A0 =A0 =A0 =A0 >
=A0 =A0 =A0 =A0 > ________________________________________
=A0 =A0 =A0 =A0 > From: lmap-bo= unces@ietf.org [lmap-bounces@i= etf.org] On Behalf Of S=F8rensen,
=A0 =A0 =A0 =A0 > Frode [frode.= sorensen@npt.no]
=A0 =A0 =A0 =A0 > Sent: 30 October 2013 14:55
=A0 =A0 =A0 =A0 > To: lmap@ietf.org=
=A0 =A0 =A0 =A0 > Subject: [lmap] LMAP regulator use case
=A0 =A0 =A0 =A0 >
=A0 =A0 =A0 =A0 > Marc, Phil, Trevor, all,
=A0 =A0 =A0 =A0 >
=A0 =A0 =A0 =A0 > I have prepared a proposed description of a regulator = use case for the
=A0 =A0 =A0 =A0 > LMAP use case ID. I noticed the "advertisement&qu= ot; for further input from
=A0 =A0 =A0 =A0 > regulators for the use case a couple of weeks ago. Fin= ally I have managed
=A0 =A0 =A0 =A0 > to find sufficient time to contribute to the ID, suppo= rted by a couple of
=A0 =A0 =A0 =A0 > "regulator friends" from the Net Neutrality = Working Group of the BEREC
=A0 =A0 =A0 =A0 > organization, representing European national regulator= s. The proposal
=A0 =A0 =A0 =A0 > consists of two parts; a modification of section 2.2 &= quot;Regulators", plus a
=A0 =A0 =A0 =A0 > new chapter after chapter 3 "Details of ISP use c= ase", titled "Details of
=A0 =A0 =A0 =A0 > regulator use case".
=A0 =A0 =A0 =A0 >
=A0 =A0 =A0 =A0 > Thanks,
=A0 =A0 =A0 =A0 > Frode
=A0 =A0 =A0 =A0 >
=A0 =A0 =A0 =A0 >
=A0 =A0 =A0 =A0 > Proposed update:
=A0 =A0 =A0 =A0 >
=A0 =A0 =A0 =A0 > 2.2 Regulators
=A0 =A0 =A0 =A0 > -------------------
=A0 =A0 =A0 =A0 > Regulators in jurisdictions around the world are respo= nding to consumers'
=A0 =A0 =A0 =A0 > adoption of Internet access services for traditional t= elecommunications
=A0 =A0 =A0 =A0 > and media services by promoting competition among prov= iders of electronic
=A0 =A0 =A0 =A0 > communications, to ensure that users derive maximum be= nefit in terms of
=A0 =A0 =A0 =A0 > choice, price, and quality.
=A0 =A0 =A0 =A0 >
=A0 =A0 =A0 =A0 > Some jurisdictions have responded to a need for greate= r information about
=A0 =A0 =A0 =A0 > Internet access service performance in the development= of regulatory
=A0 =A0 =A0 =A0 > policies and approaches for broadband technologies by = developing large-
=A0 =A0 =A0 =A0 > scale measurement programs. Programs such as the U.S. = Federal
=A0 =A0 =A0 =A0 > Communications Commission's Measuring Broadband Am= erica, European
=A0 =A0 =A0 =A0 > Commission's Quality of Broadband Services in the = EU reports and a growing
=A0 =A0 =A0 =A0 > list of other programs employ a diverse set of operati= onal and technical
=A0 =A0 =A0 =A0 > approaches to gathering data to perform analysis and r= eporting on diverse
=A0 =A0 =A0 =A0 > aspects of broadband performance.
=A0 =A0 =A0 =A0 >
=A0 =A0 =A0 =A0 > While each jurisdiction responds to distinct consumer,= industry, and
=A0 =A0 =A0 =A0 > regulatory concerns, much commonality exists in the ne= ed to produce
=A0 =A0 =A0 =A0 > datasets that are able to compare multiple Internet ac= cess service
=A0 =A0 =A0 =A0 > providers, diverse technical solutions, geographic and= regional
=A0 =A0 =A0 =A0 > distributions, and marketed and provisioned levels and= combinations of
=A0 =A0 =A0 =A0 > broadband Internet access services. In some jurisdicti= ons, the role of
=A0 =A0 =A0 =A0 > measuring is provided by a measurement provider.
=A0 =A0 =A0 =A0 >
=A0 =A0 =A0 =A0 > Measurement providers measure network performance from= users towards
=A0 =A0 =A0 =A0 > multiple content and application providers, included d= edicated test
=A0 =A0 =A0 =A0 > measurement servers, to show a performance of the actu= al Internet access
=A0 =A0 =A0 =A0 > service provided by different ISPs. Users need to know= the performance
=A0 =A0 =A0 =A0 > that are achieving from their own ISP. In addition, th= ey need to know the
=A0 =A0 =A0 =A0 > performance of other ISPs of same location as backgrou= nd information for
=A0 =A0 =A0 =A0 > selecting their ISP. Measurement providers will provid= e measurement
=A0 =A0 =A0 =A0 > results with associated measurement methods and measur= ement metrics.
=A0 =A0 =A0 =A0 >
=A0 =A0 =A0 =A0 > From a consumer perspective, the differentiation betwe= en fixed and mobile
=A0 =A0 =A0 =A0 > (cellular) Internet access services is blurring as the= applications used
=A0 =A0 =A0 =A0 > are very similar. Hence, similar measurements will tak= e place on both
=A0 =A0 =A0 =A0 > fixed and mobile Internet access services.
=A0 =A0 =A0 =A0 >
=A0 =A0 =A0 =A0 > Regulators role in the development and enforcement of = broadband Internet
=A0 =A0 =A0 =A0 > access service policies also require that the measurem= ent approaches meet
=A0 =A0 =A0 =A0 > a high level of verifiability, accuracy and provider-i= ndependence to
=A0 =A0 =A0 =A0 > support valid and meaningful comparisons of Internet a= ccess service
=A0 =A0 =A0 =A0 > performance
=A0 =A0 =A0 =A0 >
=A0 =A0 =A0 =A0 > LMAP standards could answer regulators shared needs by= providing scalable,
=A0 =A0 =A0 =A0 > cost-effective, scientifically robust solutions to the= measurement and
=A0 =A0 =A0 =A0 > collection of broadband Internet access service perfor= mance information.
=A0 =A0 =A0 =A0 >
=A0 =A0 =A0 =A0 >
=A0 =A0 =A0 =A0 > Proposed new chapter:
=A0 =A0 =A0 =A0 >
=A0 =A0 =A0 =A0 > 4 Details of regulator use case
=A0 =A0 =A0 =A0 > ---------------------------------------
=A0 =A0 =A0 =A0 > 4.1 Promoting competition through transparency
=A0 =A0 =A0 =A0 > ------------------------------------------------------= ---------
=A0 =A0 =A0 =A0 > Competition plays a vital role in regulation of the el= ectronic
=A0 =A0 =A0 =A0 > communications markets. For competition to successfull= y discipline
=A0 =A0 =A0 =A0 > operators' behaviour in the interests of their cus= tomers, end users must
=A0 =A0 =A0 =A0 > be fully aware of the characteristics of the ISPs'= access offers. In some
=A0 =A0 =A0 =A0 > jurisdictions regulators mandate transparent informati= on made available
=A0 =A0 =A0 =A0 > about service offers.
=A0 =A0 =A0 =A0 >
=A0 =A0 =A0 =A0 > End users need effective transparency to be able to ma= ke informed choices
=A0 =A0 =A0 =A0 > throughout the different stages of their relationship = with ISPs, when
=A0 =A0 =A0 =A0 > selecting Internet access service offers, and when con= sidering switching
=A0 =A0 =A0 =A0 > service offer within an ISP or to an alternative ISP. = Quality information
=A0 =A0 =A0 =A0 > about service offers can be provided based on performa= nce measurements of
=A0 =A0 =A0 =A0 > the services provided in the market.
=A0 =A0 =A0 =A0 >
=A0 =A0 =A0 =A0 > A set of measurement parameters and associated measure= ment methods are
=A0 =A0 =A0 =A0 > used over time, e.g. speed, delay, and jitter. Then th= e measurement raw
=A0 =A0 =A0 =A0 > data are collected and go through statistical post-pro= cessing before the
=A0 =A0 =A0 =A0 > results can be published in an Internet access service= quality index to
=A0 =A0 =A0 =A0 > facilitate end users' choice of service provider a= nd offer.
=A0 =A0 =A0 =A0 >
=A0 =A0 =A0 =A0 > In order to make information on offerings more meaning= ful and comparable
=A0 =A0 =A0 =A0 > to end users, common frames of reference on Internet a= ccess service
=A0 =A0 =A0 =A0 > quality metrics are essential. An example would be tha= t in relation to
=A0 =A0 =A0 =A0 > access speed, the actual download and upload speeds ar= e measured and
=A0 =A0 =A0 =A0 > published, not only the maximum speed.
=A0 =A0 =A0 =A0 >
=A0 =A0 =A0 =A0 > A measurement system that monitor Internet access serv= ices and collect
=A0 =A0 =A0 =A0 > quality information can typically consist of a number = of measurement
=A0 =A0 =A0 =A0 > probes and one or more test servers located at peering= points. The system
=A0 =A0 =A0 =A0 > can be operated by a regulator or a measurement provid= er. =A0Number and
=A0 =A0 =A0 =A0 > distribution of probes follows specific requirements d= epending on the
=A0 =A0 =A0 =A0 > scope and the desired statistical reliability of the m= easurement campaign.
=A0 =A0 =A0 =A0 >
=A0 =A0 =A0 =A0 > In many test configurations, the measurement results w= ill only reflect the
=A0 =A0 =A0 =A0 > performance of ISP's own network. However, the qua= lity of an Internet
=A0 =A0 =A0 =A0 > access service is also determined by the connectivity = to the rest of the
=A0 =A0 =A0 =A0 > Internet. Therefore test configurations which measure = beyond the ISP's own
=A0 =A0 =A0 =A0 > network may also be relevant in some cases.
=A0 =A0 =A0 =A0 >
=A0 =A0 =A0 =A0 >
=A0 =A0 =A0 =A0 > 4.2 Promoting end user empowerment
=A0 =A0 =A0 =A0 > --------------------------------------------------
=A0 =A0 =A0 =A0 > In addition to the advantages that end users receive f= rom the competition
=A0 =A0 =A0 =A0 > in the market, through availability of a plethora of I= SPs and Internet
=A0 =A0 =A0 =A0 > access service offers, facilitated by transparent qual= ity information
=A0 =A0 =A0 =A0 > about these offers at a general level, end users would= further leverage on
=A0 =A0 =A0 =A0 > information about specific services they subscribe to,= or which they are
=A0 =A0 =A0 =A0 > considering to subscribe to.
=A0 =A0 =A0 =A0 >
=A0 =A0 =A0 =A0 > Performance of Internet access service offers vary bas= ed on where the
=A0 =A0 =A0 =A0 > service is provided (geographical and topological aspe= cts), how it is
=A0 =A0 =A0 =A0 > provided (e.g. access technology and ISP's interco= nnection to the
=A0 =A0 =A0 =A0 > Internet) and when it is provided (variation through t= he day and over the
=A0 =A0 =A0 =A0 > week). Therefore there is a need to obtain more detail= ed quality
=A0 =A0 =A0 =A0 > information about specific service offers.
=A0 =A0 =A0 =A0 >
=A0 =A0 =A0 =A0 > Quality measurement tools can be made available for en= d users to monitor
=A0 =A0 =A0 =A0 > their Internet access service, giving information abou= t the speed and
=A0 =A0 =A0 =A0 > other quality metrics of the subscribed service. Speed= meters and similar
=A0 =A0 =A0 =A0 > tools can be provided by regulators or other third par= ties, but also
=A0 =A0 =A0 =A0 > directly by the ISP for the benefit of the subscriber.=
=A0 =A0 =A0 =A0 >
=A0 =A0 =A0 =A0 > Such measurement tools assist end users in verifying t= he performance of
=A0 =A0 =A0 =A0 > the Internet access service provided, comparing the me= asurement results
=A0 =A0 =A0 =A0 > achieved with the information given in the contract. I= f the contracted
=A0 =A0 =A0 =A0 > service quality level is not met, end users can use th= ese results when
=A0 =A0 =A0 =A0 > complaining to the ISP to achieve improvement and comp= ensation.
=A0 =A0 =A0 =A0 >
=A0 =A0 =A0 =A0 > It may also help end users to assess whether they have= chosen the package
=A0 =A0 =A0 =A0 > which best suits their specific needs. With their perf= ormance results at
=A0 =A0 =A0 =A0 > hand, they can benefit from quality evaluation guides,= which offer
=A0 =A0 =A0 =A0 > comparisons based on their specific usage profiles. Th= is may enable them
=A0 =A0 =A0 =A0 > to check whether their current subscription is the mos= t appropriate, and
=A0 =A0 =A0 =A0 > assist them to switch package or provider if it is in = their best interest.
=A0 =A0 =A0 =A0 >
=A0 =A0 =A0 =A0 > When provided by regulators and other third parties, s= uch quality
=A0 =A0 =A0 =A0 > measurement tools can also be used to compare performa= nce between ISPs. It
=A0 =A0 =A0 =A0 > should, though, be recognised that measurement results= can be affected by
=A0 =A0 =A0 =A0 > factors outside the ISP's control, related for ins= tance with local network
=A0 =A0 =A0 =A0 > connection (e.g. Wi-Fi), equipment and software being = used.
=A0 =A0 =A0 =A0 >
=A0 =A0 =A0 =A0 >
=A0 =A0 =A0 =A0 > 4.3 Monitoring overall market development
=A0 =A0 =A0 =A0 > ------------------------------------------------------= ---
=A0 =A0 =A0 =A0 > Governments sometimes set strategic goals for high-spe= ed Internet access
=A0 =A0 =A0 =A0 > penetration as an important component of the economic,= cultural and social
=A0 =A0 =A0 =A0 > development of the society. To evaluate the effect of = the stimulated
=A0 =A0 =A0 =A0 > growth over time, broadband Internet access take-up an= d penetration of
=A0 =A0 =A0 =A0 > high-speed access can be monitored through measurement= campaigns.
=A0 =A0 =A0 =A0 >
=A0 =A0 =A0 =A0 > An example of such an initiative is the "Digital = Agenda for Europe" which
=A0 =A0 =A0 =A0 > was adopted in 2010, to achieve universal broadband ac= cess. The goal is to
=A0 =A0 =A0 =A0 > achieve by 2020, access for all Europeans to Internet = access speeds of 30
=A0 =A0 =A0 =A0 > Mbps or above, and 50% or more of European households = subscribing to
=A0 =A0 =A0 =A0 > Internet connections above 100 Mbps.
=A0 =A0 =A0 =A0 >
=A0 =A0 =A0 =A0 > To monitor actual broadband Internet access performanc= e in a specific
=A0 =A0 =A0 =A0 > country or a region, extensive measurement campaigns a= re needed. A panel
=A0 =A0 =A0 =A0 > can be built based on operators and packages in the ma= rket, spread over
=A0 =A0 =A0 =A0 > urban, suburban and rural areas. Probes can then be di= stributed to the
=A0 =A0 =A0 =A0 > participants of the campaign.
=A0 =A0 =A0 =A0 >
=A0 =A0 =A0 =A0 > Periodic tests running on the probes can for example m= easure actual speed
=A0 =A0 =A0 =A0 > at peak and off-peak hours, but also other detailed qu= ality metrics like
=A0 =A0 =A0 =A0 > delay and jitter. Collected data goes afterwards throu= gh statistical
=A0 =A0 =A0 =A0 > analysis, deriving estimates for the whole population = which can then be
=A0 =A0 =A0 =A0 > presented and published regularly.
=A0 =A0 =A0 =A0 >
=A0 =A0 =A0 =A0 > Using a harmonized or standardised measurement methodo= logy, or even a
=A0 =A0 =A0 =A0 > common quality measurement platform, measurement resul= ts could also be
=A0 =A0 =A0 =A0 > used for benchmarking of providers and/or countries. =A0 =A0 =A0 =A0 >
=A0 =A0 =A0 =A0 >
=A0 =A0 =A0 =A0 > 4.4 Detecting degradation of the Internet access
=A0 =A0 =A0 =A0 > ------------------------------------------------------= --------
=A0 =A0 =A0 =A0 > Regulation related to net neutrality and the open Inte= rnet has been
=A0 =A0 =A0 =A0 > introduced in some jurisdictions' Internet policy,= and monitoring methods
=A0 =A0 =A0 =A0 > for detection of potential net neutrality breaches is = needed to follow up
=A0 =A0 =A0 =A0 > such regulations with concrete actions.
=A0 =A0 =A0 =A0 >
=A0 =A0 =A0 =A0 >
=A0 =A0 =A0 =A0 > Net neutrality is typically related to equal treatment= of traffic
=A0 =A0 =A0 =A0 > transmitted over the Internet access service, while ot= her traffic
=A0 =A0 =A0 =A0 > transmitted in parallel over the end user's broadb= and connection e.g. to
=A0 =A0 =A0 =A0 > provide IPTV in IP networks separated from the Interne= t (also referred to
=A0 =A0 =A0 =A0 > as "closed IP networks") can be exempted fro= m net neutrality regulation.
=A0 =A0 =A0 =A0 > Services provided in parallel to the Internet access s= ervices are referred
=A0 =A0 =A0 =A0 > to as specialised services (ref. BEREC and FCC).
=A0 =A0 =A0 =A0 >
=A0 =A0 =A0 =A0 > In order for such regulatory service architecture to p= rotect net
=A0 =A0 =A0 =A0 > neutrality and the open Internet, it is essential that= the specialised
=A0 =A0 =A0 =A0 > services are not provided at the expense of the Intern= et access service.
=A0 =A0 =A0 =A0 > Therefore regulators may need to measure performance o= f the Internet
=A0 =A0 =A0 =A0 > access service over time, and thereby detect whether t= his service is
=A0 =A0 =A0 =A0 > becoming degraded.
=A0 =A0 =A0 =A0 >
=A0 =A0 =A0 =A0 > When regulators monitor this development to ensure net= neutrality and the
=A0 =A0 =A0 =A0 > open Internet, they can check e.g. whether Internet ac= cess service
=A0 =A0 =A0 =A0 > performance and level of quality reflect advances in t= echnology and that
=A0 =A0 =A0 =A0 > this is not impaired by specialised services. Comparis= on between ISPs or
=A0 =A0 =A0 =A0 > between different countries may also be relevant for t= his kind of
=A0 =A0 =A0 =A0 > evaluation.
=A0 =A0 =A0 =A0 >
=A0 =A0 =A0 =A0 > Based on such Internet access service quality monitori= ng, regulators will
=A0 =A0 =A0 =A0 > typically report results and findings regularly, e.g. = on an annual basis.
=A0 =A0 =A0 =A0 > In some jurisdictions regulators may also eventually i= mpose minimum
=A0 =A0 =A0 =A0 > quality of service requirements on ISPs in order to pr= event degradation of
=A0 =A0 =A0 =A0 > Internet access services.
=A0 =A0 =A0 =A0 >
=A0 =A0 =A0 =A0 >
=A0 =A0 =A0 =A0 > 4.5 Detecting degradation of specific applications
=A0 =A0 =A0 =A0 > ------------------------------------------------------= ---------
=A0 =A0 =A0 =A0 > In the regulatory context related to net neutrality an= d the open Internet,
=A0 =A0 =A0 =A0 > in addition to the monitoring to detect potential degr= adation of the
=A0 =A0 =A0 =A0 > access service as a whole described above, it is essen= tial to perform
=A0 =A0 =A0 =A0 > monitoring to detect potential degradation of individu= al applications
=A0 =A0 =A0 =A0 > using the Internet access.
=A0 =A0 =A0 =A0 >
=A0 =A0 =A0 =A0 > Since net neutrality relates to equal treatment of tra= ffic transmitted
=A0 =A0 =A0 =A0 > over the Internet access service, the treatment of dif= ferent applications'
=A0 =A0 =A0 =A0 > traffic is in the core of net neutrality. When traffic= originating from
=A0 =A0 =A0 =A0 > different applications are treated independent of whic= h application each
=A0 =A0 =A0 =A0 > IP packet belongs to, this is referred to as applicati= on agnosticism.
=A0 =A0 =A0 =A0 >
=A0 =A0 =A0 =A0 > Examples of departure from application agnosticism are= blocking or
=A0 =A0 =A0 =A0 > throttling of traffic from specific applications, but = also preferential
=A0 =A0 =A0 =A0 > treatment of specific applications. If traffic is forw= arded at different
=A0 =A0 =A0 =A0 > priority levels, traffic having higher priority level = implies that other
=A0 =A0 =A0 =A0 > traffic has lower priority. Detection of such traffic = management practices
=A0 =A0 =A0 =A0 > does not necessarily imply breaches to net neutrality,= but these
=A0 =A0 =A0 =A0 > observations can be used as an input to the regulatory= evaluation of the
=A0 =A0 =A0 =A0 > practice.
=A0 =A0 =A0 =A0 >
=A0 =A0 =A0 =A0 > When regulators monitor traffic management practices t= o ensure net
=A0 =A0 =A0 =A0 > neutrality and the open Internet, they can check compl= iance with criteria
=A0 =A0 =A0 =A0 > defining what is considered reasonable traffic managem= ent. Measurement
=A0 =A0 =A0 =A0 > systems designed to monitor this kind of effects need = to send application-
=A0 =A0 =A0 =A0 > specific traffic and then measure in detail the behavi= our of the different
=A0 =A0 =A0 =A0 > packets receive when transferred over the Internet acc= ess service.
=A0 =A0 =A0 =A0 >
=A0 =A0 =A0 =A0 > Today there exist measurement tools that can detect di= fferentiated
=A0 =A0 =A0 =A0 > treatment of individual applications. Another method t= hat can be used is
=A0 =A0 =A0 =A0 > to measure performance at application layer, e.g. assi= sted by content and
=A0 =A0 =A0 =A0 > application providers, but also passive measurements a= re foreseen as a
=A0 =A0 =A0 =A0 > promising approach for monitoring of application-speci= fic treatment.
=A0 =A0 =A0 =A0 >
=A0 =A0 =A0 =A0 > _______________________________________________
=A0 =A0 =A0 =A0 > lmap mailing list
=A0 =A0 =A0 =A0 > lmap@ietf.org
=A0 =A0 =A0 =A0 > https://www.ietf.org/mailman/listinfo/lmap
=A0 =A0 =A0 =A0 > _______________________________________________
=A0 =A0 =A0 =A0 > lmap mailing list
=A0 =A0 =A0 =A0 > lmap@ietf.org
=A0 =A0 =A0 =A0 > https://www.ietf.org/mailman/listinfo/lmap
=A0 =A0 =A0 =A0 _______________________________________________
=A0 =A0 =A0 =A0 lmap mailing list
=A0 =A0 =A0 =A0 lmap@ietf.org
=A0 =A0 =A0 =A0 https://www.ietf.org/mailman/listinfo/lmap



--001a11c23ed2339faa04ea978dac-- From bs7652@att.com Thu Nov 7 13:27:41 2013 Return-Path: X-Original-To: lmap@ietfa.amsl.com Delivered-To: lmap@ietfa.amsl.com Received: from localhost (localhost [127.0.0.1]) by ietfa.amsl.com (Postfix) with ESMTP id AE3C711E8101 for ; Thu, 7 Nov 2013 13:27:37 -0800 (PST) X-Virus-Scanned: amavisd-new at amsl.com X-Spam-Flag: NO X-Spam-Score: -6.599 X-Spam-Level: X-Spam-Status: No, score=-6.599 tagged_above=-999 required=5 tests=[BAYES_00=-2.599, RCVD_IN_DNSWL_MED=-4] Received: from mail.ietf.org ([12.22.58.30]) by localhost (ietfa.amsl.com [127.0.0.1]) (amavisd-new, port 10024) with ESMTP id MHNZiSmmq2FW for ; Thu, 7 Nov 2013 13:27:31 -0800 (PST) Received: from nbfkord-smmo06.seg.att.com (nbfkord-smmo06.seg.att.com [209.65.160.94]) by ietfa.amsl.com (Postfix) with ESMTP id 7E42721E80DB for ; Thu, 7 Nov 2013 13:27:28 -0800 (PST) Received: from unknown [144.160.229.23] (EHLO nbfkord-smmo06.seg.att.com) by nbfkord-smmo06.seg.att.com(mxl_mta-6.15.0-1) with ESMTP id 0c50c725.56d59940.5877237.00-533.16496731.nbfkord-smmo06.seg.att.com (envelope-from ); Thu, 07 Nov 2013 21:27:28 +0000 (UTC) X-MXL-Hash: 527c05c067a507c7-65cd5b40ea9c83fa5be2ca896ce36a46a8d0cdbe Received: from unknown [144.160.229.23] (EHLO alpi154.enaf.aldc.att.com) by nbfkord-smmo06.seg.att.com(mxl_mta-6.15.0-1) over TLS secured channel with ESMTP id eb50c725.0.5877222.00-463.16496681.nbfkord-smmo06.seg.att.com (envelope-from ); Thu, 07 Nov 2013 21:27:27 +0000 (UTC) X-MXL-Hash: 527c05bf5f978647-5f79936387e132669fc8566b1bf6e887ca9c191e Received: from enaf.aldc.att.com (localhost [127.0.0.1]) by alpi154.enaf.aldc.att.com (8.14.5/8.14.5) with ESMTP id rA7LRP9B009180; Thu, 7 Nov 2013 16:27:26 -0500 Received: from alpi133.aldc.att.com (alpi133.aldc.att.com [130.8.217.3]) by alpi154.enaf.aldc.att.com (8.14.5/8.14.5) with ESMTP id rA7LRNCd009164 (version=TLSv1/SSLv3 cipher=DHE-RSA-AES256-SHA bits=256 verify=NO); Thu, 7 Nov 2013 16:27:24 -0500 Received: from GAALPA1MSGHUB9E.ITServices.sbc.com (GAALPA1MSGHUB9E.itservices.sbc.com [130.8.36.91]) by alpi133.aldc.att.com (RSA Interceptor); Thu, 7 Nov 2013 21:27:12 GMT Received: from GAALPA1MSGUSR9L.ITServices.sbc.com ([130.8.36.69]) by GAALPA1MSGHUB9E.ITServices.sbc.com ([130.8.36.91]) with mapi id 14.03.0158.001; Thu, 7 Nov 2013 16:27:12 -0500 From: "STARK, BARBARA H" To: "Romascanu, Dan (Dan)" , "lmap@ietf.org" Thread-Topic: [lmap] call for consensus - adoption of draft-burbridge-lmap-information-model as WG document Thread-Index: Ac7bVGevh+4pI7FlSz+236qyIybWMQAqi4Vw Date: Thu, 7 Nov 2013 21:27:11 +0000 Message-ID: <2D09D61DDFA73D4C884805CC7865E611303973A1@GAALPA1MSGUSR9L.ITServices.sbc.com> References: <9904FB1B0159DA42B0B887B7FA8119CA1292EAA3@AZ-FFEXMB04.global.avaya.com> In-Reply-To: <9904FB1B0159DA42B0B887B7FA8119CA1292EAA3@AZ-FFEXMB04.global.avaya.com> Accept-Language: en-US Content-Language: en-US X-MS-Has-Attach: X-MS-TNEF-Correlator: x-originating-ip: [135.70.3.4] Content-Type: text/plain; charset="us-ascii" Content-Transfer-Encoding: quoted-printable MIME-Version: 1.0 X-RSA-Inspected: yes X-RSA-Classifications: public X-Spam: [F=0.2000000000; CM=0.500; S=0.200(2010122901)] X-MAIL-FROM: X-SOURCE-IP: [144.160.229.23] X-AnalysisOut: [v=2.0 cv=TJDyvSZa c=1 sm=0 a=VXHOiMMwGAwA+y4G3/O+aw==:17 a] X-AnalysisOut: [=wtP7dwIN5oMA:10 a=ZpYzlOpLXHkA:10 a=ofMgfj31e3cA:10 a=BLc] X-AnalysisOut: [eEmwcHowA:10 a=kj9zAlcOel0A:10 a=zQP7CpKOAAAA:8 a=XIqpo32R] X-AnalysisOut: [AAAA:8 a=xNBSxLsk1l0A:10 a=48vgC7mUAAAA:8 a=MAG6wGB7dClMao] X-AnalysisOut: [zfvVsA:9 a=CjuIK1q_8ugA:10 a=lZB815dzVvQA:10] Subject: Re: [lmap] call for consensus - adoption of draft-burbridge-lmap-information-model as WG document X-BeenThere: lmap@ietf.org X-Mailman-Version: 2.1.12 Precedence: list List-Id: Large Scale Measurement of Access network Performance List-Unsubscribe: , List-Archive: List-Post: List-Help: List-Subscribe: , X-List-Received-Date: Thu, 07 Nov 2013 21:27:42 -0000 Sorry I couldn't be there in person or participate in the discussion live. After listening to the recorded session and carefully considering the voice= d objections (which I would paraphrase as being along the line of: we shoul= d finish use cases before taking on a new WG work item; we already see loss= of energy in doing use cases and would perhaps see a similar loss in energ= y on this if it's adopted now), I would like to see this adopted as a WG dr= aft. I believe the framework and information model are closely linked and I woul= d like to see them worked at the same time. Working them simultaneously, IM= O, can be very beneficial, which for me outweighs the potential described r= isks. Barbara > -----Original Message----- > From: lmap-bounces@ietf.org [mailto:lmap-bounces@ietf.org] On Behalf Of > Romascanu, Dan (Dan) > Sent: Wednesday, November 06, 2013 7:58 PM > To: lmap@ietf.org > Subject: [lmap] call for consensus - adoption of draft-burbridge-lmap- > information-model as WG document >=20 >=20 > Today in the LMAP meeting we ran a hum on the question whether we > should adopt draft-burbridge-lmap-information-model. The result was read > as showing consensus in favor of the adoption, although there was at leas= t > one opposed opinion expressed by Benoit (as contributor). >=20 > We would like to extend this call to the participants who could not atten= d the > meeting. Please send your opinions on this question to the WG mail list > before Friday 11/15. >=20 > Thanks and Regards, >=20 > Dan >=20 >=20 >=20 > _______________________________________________ > lmap mailing list > lmap@ietf.org > https://www.ietf.org/mailman/listinfo/lmap From Michael.K.Bugenhagen@centurylink.com Mon Nov 11 09:05:49 2013 Return-Path: X-Original-To: lmap@ietfa.amsl.com Delivered-To: lmap@ietfa.amsl.com Received: from localhost (localhost [127.0.0.1]) by ietfa.amsl.com (Postfix) with ESMTP id D172F11E8126 for ; Mon, 11 Nov 2013 09:05:49 -0800 (PST) X-Virus-Scanned: amavisd-new at amsl.com X-Spam-Flag: NO X-Spam-Score: -2.599 X-Spam-Level: X-Spam-Status: No, score=-2.599 tagged_above=-999 required=5 tests=[AWL=0.000, BAYES_00=-2.599] Received: from mail.ietf.org ([12.22.58.30]) by localhost (ietfa.amsl.com [127.0.0.1]) (amavisd-new, port 10024) with ESMTP id p0de2ra8KRM3 for ; Mon, 11 Nov 2013 09:05:41 -0800 (PST) Received: from suomp64i.qwest.com (suomp64i.qwest.com [155.70.16.237]) by ietfa.amsl.com (Postfix) with ESMTP id ED32311E8120 for ; Mon, 11 Nov 2013 09:05:40 -0800 (PST) Received: from lxdenvmpc030.qintra.com (emailout.qintra.com [10.1.51.30]) by suomp64i.qwest.com (8.14.4/8.14.4) with ESMTP id rABH5VAH015813 (version=TLSv1/SSLv3 cipher=DHE-RSA-AES256-SHA bits=256 verify=NO); Mon, 11 Nov 2013 11:05:31 -0600 (CST) Received: from lxdenvmpc030.qintra.com (unknown [127.0.0.1]) by IMSA (Postfix) with ESMTP id 6382C1E004E; Mon, 11 Nov 2013 10:05:26 -0700 (MST) Received: from suomp61i.qintra.com (unknown [151.119.91.93]) by lxdenvmpc030.qintra.com (Postfix) with ESMTP id 256921E0055; Mon, 11 Nov 2013 10:05:26 -0700 (MST) Received: from suomp61i.qintra.com (localhost [127.0.0.1]) by suomp61i.qintra.com (8.14.4/8.14.4) with ESMTP id rABH5PFI019977; Mon, 11 Nov 2013 11:05:25 -0600 (CST) Received: from vodcwhubex502.ctl.intranet (vodcwhubex502.ctl.intranet [151.117.206.28]) by suomp61i.qintra.com (8.14.4/8.14.4) with ESMTP id rABH5PVu019954 (version=TLSv1/SSLv3 cipher=AES128-SHA bits=128 verify=FAIL); Mon, 11 Nov 2013 11:05:25 -0600 (CST) Received: from PODCWMBXEX505.ctl.intranet ([fe80::f87e:fe44:ad72:b610]) by vodcwhubex502.ctl.intranet ([2002:9775:ce1c::9775:ce1c]) with mapi id 14.02.0318.001; Mon, 11 Nov 2013 11:05:24 -0600 From: "Bugenhagen, Michael K" To: "'Romascanu, Dan (Dan)'" , "lmap@ietf.org" , "'Trevor Burbridge'" , "'philip.eardley@bt.com'" , "'marcelo bagnulo braun'" Thread-Topic: Comments & opinions on Draft - RE: [lmap] call for consensus - adoption of draft-burbridge-lmap-information-model as WG document Thread-Index: Ac7bVGevh+4pI7FlSz+236qyIybWMQDqGOVA Date: Mon, 11 Nov 2013 17:05:24 +0000 Message-ID: References: <9904FB1B0159DA42B0B887B7FA8119CA1292EAA3@AZ-FFEXMB04.global.avaya.com> In-Reply-To: <9904FB1B0159DA42B0B887B7FA8119CA1292EAA3@AZ-FFEXMB04.global.avaya.com> Accept-Language: en-US Content-Language: en-US X-MS-Has-Attach: X-MS-TNEF-Correlator: x-originating-ip: [151.117.206.7] Content-Type: text/plain; charset="us-ascii" Content-Transfer-Encoding: quoted-printable MIME-Version: 1.0 X-CFilter-Loop: Reflected Cc: "'Stark, Barbara'" , "Cook, Charles" , "'MORTON, ALFRED C \(AL\)'" , 'KEN KO' Subject: [lmap] Comments & opinions on Draft - RE: call for consensus - adoption of draft-burbridge-lmap-information-model as WG document X-BeenThere: lmap@ietf.org X-Mailman-Version: 2.1.12 Precedence: list List-Id: Large Scale Measurement of Access network Performance List-Unsubscribe: , List-Archive: List-Post: List-Help: List-Subscribe: , X-List-Received-Date: Mon, 11 Nov 2013 17:05:50 -0000 Dan,=20 Comment 1. Is written from an ISP Operational team perspective (Scope says = ISP use case so... ) Comment 2. Is future proofing the Registry concept but it also is in the dr= aft a bit.. 1) ISP use case requirements - (The yet another test system overlay issue) ISP's have multiple IP based "testing systems" / "Measurement Agents" dep= loyed today, many of which don't co-exist well. So one ongoing fairly soli= d ISP requirement that the groups should strongly consider is the ability o= f a MA to be open flexible framework MA that can run both Op's ADHOC test, = and performance testing without two MA's. Example (a MA / probe that can r= un VoIP, VIDEO, LMAP, DNS testing, ..... vs. multiple probes doing one each= .) ** Current draft that means adding the ability to=20 ADHOC / Multi-user op's needs for considerations=20 a) MA capability to both Push and Pull test results=20 b) MA Radius / Authentication to allow item a) c) Multiple test capability (OAM, and throughput at the same time by differ= ent Op's teams) d) Resource availability checking (when you can run more than one test at a= time) e) MA to MA test capability negotiation (if you are working with a customer= s or 3rd party probe) 2) Future proof the Registry / MA probe concept into Virtual Applications -= (start thinking Virtualization Service Catalog)=09 As NFV (network functional virtualization) will likely become available= by the time some of this is being implemented we should consider adding a = "hypervisor" type to the Registry. The thought here is that Virtual applic= ations come in a default deployable set, *.OVF files for VMware, ... for KV= M, and so on. ** ONE KEY to service catalogs is a trusted source for the= code, one should NEVER (NEVER - EVER) deploy a virtual appliance from a pu= blic source on their network...=20 =20 I think the registry could be a good solution to this issue - provided we p= lace a high level or safety checks on anything getting loaded into the regi= stry itself. -----Original Message----- From: lmap-bounces@ietf.org [mailto:lmap-bounces@ietf.org] On Behalf Of Rom= ascanu, Dan (Dan) Sent: Wednesday, November 06, 2013 6:58 PM To: lmap@ietf.org Subject: [lmap] call for consensus - adoption of draft-burbridge-lmap-infor= mation-model as WG document Today in the LMAP meeting we ran a hum on the question whether we should ad= opt draft-burbridge-lmap-information-model. The result was read as showing = consensus in favor of the adoption, although there was at least one opposed= opinion expressed by Benoit (as contributor).=20 We would like to extend this call to the participants who could not attend = the meeting. Please send your opinions on this question to the WG mail list= before Friday 11/15.=20 Thanks and Regards, Dan _______________________________________________ lmap mailing list lmap@ietf.org https://www.ietf.org/mailman/listinfo/lmap From steve@idrathernotsay.com Mon Nov 11 10:29:00 2013 Return-Path: X-Original-To: lmap@ietfa.amsl.com Delivered-To: lmap@ietfa.amsl.com Received: from localhost (localhost [127.0.0.1]) by ietfa.amsl.com (Postfix) with ESMTP id 1872111E8153 for ; Mon, 11 Nov 2013 10:29:00 -0800 (PST) X-Virus-Scanned: amavisd-new at amsl.com X-Spam-Flag: NO X-Spam-Score: -1.392 X-Spam-Level: X-Spam-Status: No, score=-1.392 tagged_above=-999 required=5 tests=[AWL=1.207, BAYES_00=-2.599] Received: from mail.ietf.org ([12.22.58.30]) by localhost (ietfa.amsl.com [127.0.0.1]) (amavisd-new, port 10024) with ESMTP id coGMpzi8aCEh for ; Mon, 11 Nov 2013 10:28:55 -0800 (PST) Received: from colo3.idrathernotsay.com (colo3.idrathernotsay.com [74.207.251.199]) by ietfa.amsl.com (Postfix) with ESMTP id 4194A21E81DA for ; Mon, 11 Nov 2013 10:28:53 -0800 (PST) Received: from localhost (localhost [127.0.0.1]) by colo3.idrathernotsay.com (Postfix) with ESMTP id 387E7FC40 for ; Mon, 11 Nov 2013 18:28:51 +0000 (UTC) X-Virus-Scanned: amavisd-new at idrathernotsay.com Received: from colo3.idrathernotsay.com ([127.0.0.1]) by localhost (colo3.idrathernotsay.com [127.0.0.1]) (amavisd-new, port 10024) with ESMTP id oF3X34aPygB9 for ; Mon, 11 Nov 2013 18:28:46 +0000 (UTC) Received: from newbie.idrathernotsay.com (pool-173-79-225-215.washdc.fios.verizon.net [173.79.225.215]) by colo3.idrathernotsay.com (Postfix) with ESMTPSA id 8454EF9BA for ; Mon, 11 Nov 2013 18:28:46 +0000 (UTC) Received: from newbie.idrathernotsay.com (localhost [127.0.0.1]) by newbie.idrathernotsay.com (Postfix) with ESMTP id 491804FBFEA for ; Mon, 11 Nov 2013 13:28:45 -0500 (EST) X-Virus-Scanned: amavisd-new at idrathernotsay.com Received: from newbie.idrathernotsay.com ([127.0.0.1]) by newbie.idrathernotsay.com (newbie.idrathernotsay.com [127.0.0.1]) (amavisd-new, port 10024) with ESMTP id SFNUcRvPmGhN for ; Mon, 11 Nov 2013 13:28:39 -0500 (EST) Received: by newbie.idrathernotsay.com (Postfix, from userid 1000) id 86BEC4FC084; Mon, 11 Nov 2013 13:28:39 -0500 (EST) Date: Mon, 11 Nov 2013 13:28:39 -0500 From: Steve Miller To: lmap@ietf.org Message-ID: <20131111182839.GA21454@idrathernotsay.com> MIME-Version: 1.0 Content-Type: text/plain; charset=us-ascii Content-Disposition: inline User-Agent: Mutt/1.5.21 (2010-09-15) Subject: [lmap] Splitting measurement tasks into measurement tasks plus reporting tasks? X-BeenThere: lmap@ietf.org X-Mailman-Version: 2.1.12 Precedence: list List-Id: Large Scale Measurement of Access network Performance List-Unsubscribe: , List-Archive: List-Post: List-Help: List-Subscribe: , X-List-Received-Date: Mon, 11 Nov 2013 18:29:00 -0000 Hi. I was thinking through some of the LMAP model, and to some extent doing so in the context of the (not entirely dissimilar) system we use within Verisign. One thing that our internal system includes is some support for alert checking and generation, as a thing that can be separate from "pure" performance reporting. I think that might be a useful capability to have in LMAP as well. In the framework, I'd think we'd represent that by breaking measurement tasks into two pieces. The first piece would define just the actual measurement task; the second piece would be a separate reporting task, which would consume the results of the measurement task. It'd be possible to feed the results of a measurement task into more than one reporting task. Let's look at the example from page 7 of http://www.ietf.org/id/draft-burbridge-lmap-information-model-01.txt: Example: A Measurement Task Configuration may configure a single Measurement Task for measuring UDP latency. The Measurement Task Configuration could define the destination port and address for the measurement as well as the duration, internal packet timing strategy and other parameters (for example a stream for one hour and sending one packet every 500 ms). It may also define the output type and possible parameters (for example the output type can be the 95th percentile mean) where the measurement task accepts such parameters. It does NOT define when the task starts (this is defined by the Measurement Schedule element), so it does not by itself instruct the MA to actually perform this measurement task. Instead of this single measurement task, we'd have the "core" measurement task: Perform the UDP latency check against X every 500ms, between the hours of 10PM and 6AM; call this check "UDP-latency-X" and then the reporting task implied above: Consume the output from measurement task "UDP-latency-X", reporting the 95th percentile mean once an hour; send that report to collector C1 and then we could also add a second reporting task: Consume the output from measurement task "UDP-latency-X", computing the exponentially-decaying average loss over the last 30 seconds. If that loss exceeds 5%, report that immediately; do so to collector C2. If the exponentially-decaying average loss over the last 30 seconds was in excess of 5% but now is less than 3%, report that immediately as well; do so to collector C2. Admittedly, for the regulator use case, I don't know how useful that is, though it seems that there would be some value in having devices that were about to note a SLA violation not wait for an extended period of time to inform the carrier that they are seeing issues. I think that'd help a lot for the separate-but-overlapping use case I'd articulated previously, in which operators of heavily anycasted infrastructure are looking for devices located in a large number of ISPs that they can tell to check their view of the perfomance of those services, so that they can know that some corner of the Internet is having issues reaching them. I don't think this'd be too complicated to implement, either. Would this be worth considering? If so, I'd be happy to draft some language around this. -Steve From j.schoenwaelder@jacobs-university.de Mon Nov 11 12:42:29 2013 Return-Path: X-Original-To: lmap@ietfa.amsl.com Delivered-To: lmap@ietfa.amsl.com Received: from localhost (localhost [127.0.0.1]) by ietfa.amsl.com (Postfix) with ESMTP id BC7E711E8112 for ; Mon, 11 Nov 2013 12:42:29 -0800 (PST) X-Virus-Scanned: amavisd-new at amsl.com X-Spam-Flag: NO X-Spam-Score: -103.197 X-Spam-Level: X-Spam-Status: No, score=-103.197 tagged_above=-999 required=5 tests=[AWL=0.052, BAYES_00=-2.599, HELO_EQ_DE=0.35, RCVD_IN_DNSWL_LOW=-1, USER_IN_WHITELIST=-100] Received: from mail.ietf.org ([12.22.58.30]) by localhost (ietfa.amsl.com [127.0.0.1]) (amavisd-new, port 10024) with ESMTP id cynisYK8f-Ob for ; Mon, 11 Nov 2013 12:42:25 -0800 (PST) Received: from hermes.jacobs-university.de (hermes.jacobs-university.de [212.201.44.23]) by ietfa.amsl.com (Postfix) with ESMTP id 0136111E8110 for ; Mon, 11 Nov 2013 12:42:25 -0800 (PST) Received: from localhost (demetrius4.jacobs-university.de [212.201.44.49]) by hermes.jacobs-university.de (Postfix) with ESMTP id 6C2452006F; Mon, 11 Nov 2013 21:42:24 +0100 (CET) X-Virus-Scanned: amavisd-new at jacobs-university.de Received: from hermes.jacobs-university.de ([212.201.44.23]) by localhost (demetrius4.jacobs-university.de [212.201.44.32]) (amavisd-new, port 10024) with ESMTP id uHoL3rmv3lUY; Mon, 11 Nov 2013 21:42:24 +0100 (CET) Received: from elstar.local (elstar.jacobs.jacobs-university.de [10.50.231.133]) by hermes.jacobs-university.de (Postfix) with ESMTP id ED7DA2006D; Mon, 11 Nov 2013 21:42:23 +0100 (CET) Received: by elstar.local (Postfix, from userid 501) id 7DF3A2939CDB; Mon, 11 Nov 2013 21:42:18 +0100 (CET) Date: Mon, 11 Nov 2013 21:42:18 +0100 From: Juergen Schoenwaelder To: Steve Miller Message-ID: <20131111204218.GB770@elstar.local> Mail-Followup-To: Steve Miller , lmap@ietf.org References: <20131111182839.GA21454@idrathernotsay.com> MIME-Version: 1.0 Content-Type: text/plain; charset=us-ascii Content-Disposition: inline In-Reply-To: <20131111182839.GA21454@idrathernotsay.com> User-Agent: Mutt/1.5.21 (2010-09-15) Cc: lmap@ietf.org Subject: Re: [lmap] Splitting measurement tasks into measurement tasks plus reporting tasks? X-BeenThere: lmap@ietf.org X-Mailman-Version: 2.1.12 Precedence: list Reply-To: Juergen Schoenwaelder List-Id: Large Scale Measurement of Access network Performance List-Unsubscribe: , List-Archive: List-Post: List-Help: List-Subscribe: , X-List-Received-Date: Mon, 11 Nov 2013 20:42:29 -0000 On Mon, Nov 11, 2013 at 01:28:39PM -0500, Steve Miller wrote: > Hi. I was thinking through some of the LMAP model, and to some extent doing so in the context of the (not entirely dissimilar) system we use within Verisign. One thing that our internal system includes is some support for alert checking and generation, as a thing that can be separate from "pure" performance reporting. I think that might be a useful capability to have in LMAP as well. > > In the framework, I'd think we'd represent that by breaking measurement tasks into two pieces. The first piece would define just the actual measurement task; the second piece would be a separate reporting task, which would consume the results of the measurement task. It'd be possible to feed the results of a measurement task into more than one reporting task. > > Let's look at the example from page 7 of http://www.ietf.org/id/draft-burbridge-lmap-information-model-01.txt: > > Example: A Measurement Task Configuration may configure a single > Measurement Task for measuring UDP latency. The Measurement Task > Configuration could define the destination port and address for > the measurement as well as the duration, internal packet timing > strategy and other parameters (for example a stream for one hour > and sending one packet every 500 ms). It may also define the > output type and possible parameters (for example the output type > can be the 95th percentile mean) where the measurement task > accepts such parameters. It does NOT define when the task starts > (this is defined by the Measurement Schedule element), so it does > not by itself instruct the MA to actually perform this measurement > task. > > Instead of this single measurement task, we'd have the "core" measurement task: > > Perform the UDP latency check against X every 500ms, between the hours of 10PM and 6AM; call this check "UDP-latency-X" > > and then the reporting task implied above: > > Consume the output from measurement task "UDP-latency-X", reporting the 95th percentile mean once an hour; send that report to collector C1 > > and then we could also add a second reporting task: > > Consume the output from measurement task "UDP-latency-X", computing the exponentially-decaying average loss over the last 30 seconds. If that loss exceeds 5%, report that immediately; do so to collector C2. If the exponentially-decaying average loss over the last 30 seconds was in excess of 5% but now is less than 3%, report that immediately as well; do so to collector C2. > > Admittedly, for the regulator use case, I don't know how useful that is, though it seems that there would be some value in having devices that were about to note a SLA violation not wait for an extended period of time to inform the carrier that they are seeing issues. I think that'd help a lot for the separate-but-overlapping use case I'd articulated previously, in which operators of heavily anycasted infrastructure are looking for devices located in a large number of ISPs that they can tell to check their view of the perfomance of those services, so that they can know that some corner of the Internet is having issues reaching them. > > I don't think this'd be too complicated to implement, either. > > Would this be worth considering? If so, I'd be happy to draft some language around this. > I hear two suggestions here: a) Give reporting channels their own independent schedule (right now they are essentially tied to the scheduled task). b) Provide a means for post-processing raw data in order to derive certain statistics before the result is shipped via a report channel. Is this a fair summary of the request? /js -- Juergen Schoenwaelder Jacobs University Bremen gGmbH Phone: +49 421 200 3587 Campus Ring 1, 28759 Bremen, Germany Fax: +49 421 200 3103 From j.schoenwaelder@jacobs-university.de Mon Nov 11 12:39:47 2013 Return-Path: X-Original-To: lmap@ietfa.amsl.com Delivered-To: lmap@ietfa.amsl.com Received: from localhost (localhost [127.0.0.1]) by ietfa.amsl.com (Postfix) with ESMTP id 0A45A11E8112 for ; Mon, 11 Nov 2013 12:39:47 -0800 (PST) X-Virus-Scanned: amavisd-new at amsl.com X-Spam-Flag: NO X-Spam-Score: -103.196 X-Spam-Level: X-Spam-Status: No, score=-103.196 tagged_above=-999 required=5 tests=[AWL=0.053, BAYES_00=-2.599, HELO_EQ_DE=0.35, RCVD_IN_DNSWL_LOW=-1, USER_IN_WHITELIST=-100] Received: from mail.ietf.org ([12.22.58.30]) by localhost (ietfa.amsl.com [127.0.0.1]) (amavisd-new, port 10024) with ESMTP id M3Wx5LsqBttx for ; Mon, 11 Nov 2013 12:39:41 -0800 (PST) Received: from hermes.jacobs-university.de (hermes.jacobs-university.de [212.201.44.23]) by ietfa.amsl.com (Postfix) with ESMTP id 855EE11E8110 for ; Mon, 11 Nov 2013 12:39:40 -0800 (PST) Received: from localhost (demetrius2.jacobs-university.de [212.201.44.47]) by hermes.jacobs-university.de (Postfix) with ESMTP id AD7EA20061; Mon, 11 Nov 2013 21:39:39 +0100 (CET) X-Virus-Scanned: amavisd-new at jacobs-university.de Received: from hermes.jacobs-university.de ([212.201.44.23]) by localhost (demetrius2.jacobs-university.de [212.201.44.32]) (amavisd-new, port 10024) with ESMTP id NhgBptF6CIQC; Mon, 11 Nov 2013 21:39:39 +0100 (CET) Received: from elstar.local (elstar.jacobs.jacobs-university.de [10.50.231.133]) by hermes.jacobs-university.de (Postfix) with ESMTP id 8A3E520058; Mon, 11 Nov 2013 21:39:38 +0100 (CET) Received: by elstar.local (Postfix, from userid 501) id 00C9D2939C64; Mon, 11 Nov 2013 21:39:32 +0100 (CET) Date: Mon, 11 Nov 2013 21:39:31 +0100 From: Juergen Schoenwaelder To: "Bugenhagen, Michael K" Message-ID: <20131111203930.GA770@elstar.local> Mail-Followup-To: "Bugenhagen, Michael K" , "'Romascanu, Dan (Dan)'" , "lmap@ietf.org" , 'Trevor Burbridge' , "'philip.eardley@bt.com'" , 'marcelo bagnulo braun' , "'Stark, Barbara'" , "Cook, Charles" , "'MORTON, ALFRED C (AL)'" , 'KEN KO' References: <9904FB1B0159DA42B0B887B7FA8119CA1292EAA3@AZ-FFEXMB04.global.avaya.com> MIME-Version: 1.0 Content-Type: text/plain; charset=us-ascii Content-Disposition: inline In-Reply-To: User-Agent: Mutt/1.5.21 (2010-09-15) X-Mailman-Approved-At: Mon, 11 Nov 2013 14:07:36 -0800 Cc: "Cook, Charles" , "'MORTON, ALFRED C \(AL\)'" , "lmap@ietf.org" , 'Trevor Burbridge' , "'Romascanu, Dan \(Dan\)'" , 'KEN KO' , 'marcelo bagnulo braun' , "'Stark, Barbara'" , "'philip.eardley@bt.com'" Subject: Re: [lmap] Comments & opinions on Draft - RE: call for consensus - adoption of draft-burbridge-lmap-information-model as WG document X-BeenThere: lmap@ietf.org X-Mailman-Version: 2.1.12 Precedence: list Reply-To: Juergen Schoenwaelder List-Id: Large Scale Measurement of Access network Performance List-Unsubscribe: , List-Archive: List-Post: List-Help: List-Subscribe: , X-List-Received-Date: Mon, 11 Nov 2013 20:39:47 -0000 On Mon, Nov 11, 2013 at 05:05:24PM +0000, Bugenhagen, Michael K wrote: > Dan, > > Comment 1. Is written from an ISP Operational team perspective (Scope says ISP use case so... ) > Comment 2. Is future proofing the Registry concept but it also is in the draft a bit.. > > > 1) ISP use case requirements - (The yet another test system overlay issue) > > ISP's have multiple IP based "testing systems" / "Measurement Agents" deployed today, many of which don't co-exist well. So one ongoing fairly solid ISP requirement that the groups should strongly consider is the ability of a MA to be open flexible framework MA that can run both Op's ADHOC test, and performance testing without two MA's. Example (a MA / probe that can run VoIP, VIDEO, LMAP, DNS testing, ..... vs. multiple probes doing one each.) > > ** Current draft that means adding the ability to > > ADHOC / Multi-user op's needs for considerations > a) MA capability to both Push and Pull test results > b) MA Radius / Authentication to allow item a) > c) Multiple test capability (OAM, and throughput at the same time by different Op's teams) > d) Resource availability checking (when you can run more than one test at a time) > e) MA to MA test capability negotiation (if you are working with a customers or 3rd party probe) > How do a)-e) follow from the use case text? > 2) Future proof the Registry / MA probe concept into Virtual Applications - (start thinking Virtualization Service Catalog) > > As NFV (network functional virtualization) will likely become available by the time some of this is being implemented we should consider adding a "hypervisor" type to the Registry. The thought here is that Virtual applications come in a default deployable set, *.OVF files for VMware, ... for KVM, and so on. ** ONE KEY to service catalogs is a trusted source for the code, one should NEVER (NEVER - EVER) deploy a virtual appliance from a public source on their network... > > I think the registry could be a good solution to this issue - provided we place a high level or safety checks on anything getting loaded into the registry itself. > The registry is about metrics. I fail to understand what change is requested here that affects the current LMAP documents. /js -- Juergen Schoenwaelder Jacobs University Bremen gGmbH Phone: +49 421 200 3587 Campus Ring 1, 28759 Bremen, Germany Fax: +49 421 200 3103 From Michael.K.Bugenhagen@centurylink.com Mon Nov 11 12:46:59 2013 Return-Path: X-Original-To: lmap@ietfa.amsl.com Delivered-To: lmap@ietfa.amsl.com Received: from localhost (localhost [127.0.0.1]) by ietfa.amsl.com (Postfix) with ESMTP id AA5E121E8098 for ; Mon, 11 Nov 2013 12:46:59 -0800 (PST) X-Virus-Scanned: amavisd-new at amsl.com X-Spam-Flag: NO X-Spam-Score: -2.599 X-Spam-Level: X-Spam-Status: No, score=-2.599 tagged_above=-999 required=5 tests=[AWL=0.000, BAYES_00=-2.599] Received: from mail.ietf.org ([12.22.58.30]) by localhost (ietfa.amsl.com [127.0.0.1]) (amavisd-new, port 10024) with ESMTP id 45h3OvD4Ot3W for ; Mon, 11 Nov 2013 12:46:53 -0800 (PST) Received: from suomp64i.qwest.com (suomp64i.qwest.com [155.70.16.237]) by ietfa.amsl.com (Postfix) with ESMTP id 7464911E811D for ; Mon, 11 Nov 2013 12:46:53 -0800 (PST) Received: from lxomavmpc030.qintra.com (lxomavmpc030.qintra.com [151.117.207.30]) by suomp64i.qwest.com (8.14.4/8.14.4) with ESMTP id rABKkg7o018040 (version=TLSv1/SSLv3 cipher=DHE-RSA-AES256-SHA bits=256 verify=NO); Mon, 11 Nov 2013 14:46:42 -0600 (CST) Received: from lxomavmpc030.qintra.com (unknown [127.0.0.1]) by IMSA (Postfix) with ESMTP id 7833B1E0053; Mon, 11 Nov 2013 14:46:37 -0600 (CST) Received: from suomp60i.qintra.com (unknown [10.6.10.61]) by lxomavmpc030.qintra.com (Postfix) with ESMTP id 56C1E1E003F; Mon, 11 Nov 2013 14:46:37 -0600 (CST) Received: from suomp60i.qintra.com (localhost [127.0.0.1]) by suomp60i.qintra.com (8.14.4/8.14.4) with ESMTP id rABKka8f026001; Mon, 11 Nov 2013 14:46:37 -0600 (CST) Received: from vodcwhubex501.ctl.intranet (vodcwhubex501.ctl.intranet [151.117.206.27]) by suomp60i.qintra.com (8.14.4/8.14.4) with ESMTP id rABKkaYJ025983 (version=TLSv1/SSLv3 cipher=AES128-SHA bits=128 verify=FAIL); Mon, 11 Nov 2013 14:46:36 -0600 (CST) Received: from PODCWMBXEX505.ctl.intranet ([fe80::f87e:fe44:ad72:b610]) by vodcwhubex501.ctl.intranet ([151.117.206.27]) with mapi id 14.02.0318.001; Mon, 11 Nov 2013 14:46:36 -0600 From: "Bugenhagen, Michael K" To: "'Juergen Schoenwaelder'" Thread-Topic: [lmap] Comments & opinions on Draft - RE: call for consensus - adoption of draft-burbridge-lmap-information-model as WG document Thread-Index: AQHO3x4nwl1kPZbVy0C+w4vn4eORgJogftNg Date: Mon, 11 Nov 2013 20:46:35 +0000 Message-ID: References: <9904FB1B0159DA42B0B887B7FA8119CA1292EAA3@AZ-FFEXMB04.global.avaya.com> <20131111203930.GA770@elstar.local> In-Reply-To: <20131111203930.GA770@elstar.local> Accept-Language: en-US Content-Language: en-US X-MS-Has-Attach: X-MS-TNEF-Correlator: x-originating-ip: [151.117.206.7] Content-Type: text/plain; charset="us-ascii" Content-Transfer-Encoding: quoted-printable MIME-Version: 1.0 X-CFilter-Loop: Reflected X-Mailman-Approved-At: Mon, 11 Nov 2013 14:07:36 -0800 Cc: "Cook, Charles" , "'MORTON, ALFRED C \(AL\)'" , "lmap@ietf.org" , 'Trevor Burbridge' , "'Romascanu, Dan \(Dan\)'" , 'KEN KO' , 'marcelo bagnulo braun' , "'Stark, Barbara'" , "'philip.eardley@bt.com'" Subject: Re: [lmap] Comments & opinions on Draft - RE: call for consensus - adoption of draft-burbridge-lmap-information-model as WG document X-BeenThere: lmap@ietf.org X-Mailman-Version: 2.1.12 Precedence: list List-Id: Large Scale Measurement of Access network Performance List-Unsubscribe: , List-Archive: List-Post: List-Help: List-Subscribe: , X-List-Received-Date: Mon, 11 Nov 2013 20:46:59 -0000 Juergen, If you look at the bigger picture my comments are saying we should consi= der those 2 items as requirements of the use cases. Or at least consider them as such... before running too far into any model = that may break if critical requirements are missed. The 2 I am suggesting here - 1) ISP's need a MA that can Stand by itself as an Overlay, but more importa= ntly they may need one that operates more globally as a universal MA. 2) If we adopt a registry and we know virtualization is on it's way, then w= e need to take that into account (a virtual MA). I gave some context around those, but the major points are what they are. Without consensus on requirements it's probably not worth trying to mash th= em into the current draft. However those two items are a bit haunting (I don't think they'll go away). Thanks, Mike -----Original Message----- From: Juergen Schoenwaelder [mailto:j.schoenwaelder@jacobs-university.de]=20 Sent: Monday, November 11, 2013 2:40 PM To: Bugenhagen, Michael K Cc: 'Romascanu, Dan (Dan)'; lmap@ietf.org; 'Trevor Burbridge'; 'philip.eard= ley@bt.com'; 'marcelo bagnulo braun'; 'Stark, Barbara'; Cook, Charles; 'MOR= TON, ALFRED C (AL)'; 'KEN KO' Subject: Re: [lmap] Comments & opinions on Draft - RE: call for consensus -= adoption of draft-burbridge-lmap-information-model as WG document On Mon, Nov 11, 2013 at 05:05:24PM +0000, Bugenhagen, Michael K wrote: > Dan, >=20 > Comment 1. Is written from an ISP Operational team perspective (Scope=20 > says ISP use case so... ) Comment 2. Is future proofing the Registry conc= ept but it also is in the draft a bit.. >=20 >=20 > 1) ISP use case requirements - (The yet another test system overlay=20 > issue) >=20 > ISP's have multiple IP based "testing systems" / "Measurement=20 > Agents" deployed today, many of which don't co-exist well. So one=20 > ongoing fairly solid ISP requirement that the groups should strongly=20 > consider is the ability of a MA to be open flexible framework MA that=20 > can run both Op's ADHOC test, and performance testing without two=20 > MA's. Example (a MA / probe that can run VoIP, VIDEO, LMAP, DNS=20 > testing, ..... vs. multiple probes doing one each.) >=20 > ** Current draft that means adding the ability to >=20 > ADHOC / Multi-user op's needs for considerations > a) MA capability to both Push and Pull test results > b) MA Radius / Authentication to allow item a) > c) Multiple test capability (OAM, and throughput at the same time by=20 > different Op's teams) > d) Resource availability checking (when you can run more than one test=20 > at a time) > e) MA to MA test capability negotiation (if you are working with a=20 > customers or 3rd party probe) >=20 How do a)-e) follow from the use case text? > 2) Future proof the Registry / MA probe concept into Virtual Applications= - (start thinking Virtualization Service Catalog)=09 >=20 > As NFV (network functional virtualization) will likely become availab= le by the time some of this is being implemented we should consider adding = a "hypervisor" type to the Registry. The thought here is that Virtual appl= ications come in a default deployable set, *.OVF files for VMware, ... for = KVM, and so on. ** ONE KEY to service catalogs is a trusted source for t= he code, one should NEVER (NEVER - EVER) deploy a virtual appliance from a = public source on their network...=20 > =20 > I think the registry could be a good solution to this issue - provided we= place a high level or safety checks on anything getting loaded into the re= gistry itself. >=20 The registry is about metrics. I fail to understand what change is requeste= d here that affects the current LMAP documents. /js --=20 Juergen Schoenwaelder Jacobs University Bremen gGmbH Phone: +49 421 200 3587 Campus Ring 1, 28759 Bremen, Germany Fax: +49 421 200 3103 From j.schoenwaelder@jacobs-university.de Mon Nov 11 13:47:31 2013 Return-Path: X-Original-To: lmap@ietfa.amsl.com Delivered-To: lmap@ietfa.amsl.com Received: from localhost (localhost [127.0.0.1]) by ietfa.amsl.com (Postfix) with ESMTP id C0CF311E8145 for ; Mon, 11 Nov 2013 13:47:27 -0800 (PST) X-Virus-Scanned: amavisd-new at amsl.com X-Spam-Flag: NO X-Spam-Score: -103.198 X-Spam-Level: X-Spam-Status: No, score=-103.198 tagged_above=-999 required=5 tests=[AWL=0.051, BAYES_00=-2.599, HELO_EQ_DE=0.35, RCVD_IN_DNSWL_LOW=-1, USER_IN_WHITELIST=-100] Received: from mail.ietf.org ([12.22.58.30]) by localhost (ietfa.amsl.com [127.0.0.1]) (amavisd-new, port 10024) with ESMTP id AcSXw6qKKTVH for ; Mon, 11 Nov 2013 13:47:22 -0800 (PST) Received: from hermes.jacobs-university.de (hermes.jacobs-university.de [212.201.44.23]) by ietfa.amsl.com (Postfix) with ESMTP id 30F6E11E80F9 for ; Mon, 11 Nov 2013 13:47:21 -0800 (PST) Received: from localhost (demetrius4.jacobs-university.de [212.201.44.49]) by hermes.jacobs-university.de (Postfix) with ESMTP id 5D93720054; Mon, 11 Nov 2013 22:47:19 +0100 (CET) X-Virus-Scanned: amavisd-new at jacobs-university.de Received: from hermes.jacobs-university.de ([212.201.44.23]) by localhost (demetrius4.jacobs-university.de [212.201.44.32]) (amavisd-new, port 10024) with ESMTP id 7mob9j3NVkt0; Mon, 11 Nov 2013 22:47:19 +0100 (CET) Received: from elstar.local (elstar.jacobs.jacobs-university.de [10.50.231.133]) by hermes.jacobs-university.de (Postfix) with ESMTP id 12F5720042; Mon, 11 Nov 2013 22:47:17 +0100 (CET) Received: by elstar.local (Postfix, from userid 501) id D4B662939DB9; Mon, 11 Nov 2013 22:47:12 +0100 (CET) Date: Mon, 11 Nov 2013 22:47:12 +0100 From: Juergen Schoenwaelder To: "Bugenhagen, Michael K" Message-ID: <20131111214712.GA977@elstar.local> Mail-Followup-To: "Bugenhagen, Michael K" , "'Romascanu, Dan (Dan)'" , "lmap@ietf.org" , 'Trevor Burbridge' , "'philip.eardley@bt.com'" , 'marcelo bagnulo braun' , "'Stark, Barbara'" , "Cook, Charles" , "'MORTON, ALFRED C (AL)'" , 'KEN KO' References: <9904FB1B0159DA42B0B887B7FA8119CA1292EAA3@AZ-FFEXMB04.global.avaya.com> <20131111203930.GA770@elstar.local> MIME-Version: 1.0 Content-Type: text/plain; charset=us-ascii Content-Disposition: inline In-Reply-To: User-Agent: Mutt/1.5.21 (2010-09-15) X-Mailman-Approved-At: Mon, 11 Nov 2013 14:07:36 -0800 Cc: "Cook, Charles" , "'MORTON, ALFRED C \(AL\)'" , "lmap@ietf.org" , 'Trevor Burbridge' , "'Romascanu, Dan \(Dan\)'" , 'KEN KO' , 'marcelo bagnulo braun' , "'Stark, Barbara'" , "'philip.eardley@bt.com'" Subject: Re: [lmap] Comments & opinions on Draft - RE: call for consensus - adoption of draft-burbridge-lmap-information-model as WG document X-BeenThere: lmap@ietf.org X-Mailman-Version: 2.1.12 Precedence: list Reply-To: Juergen Schoenwaelder List-Id: Large Scale Measurement of Access network Performance List-Unsubscribe: , List-Archive: List-Post: List-Help: List-Subscribe: , X-List-Received-Date: Mon, 11 Nov 2013 21:47:31 -0000 On Mon, Nov 11, 2013 at 08:46:35PM +0000, Bugenhagen, Michael K wrote: > Juergen, > > If you look at the bigger picture my comments are saying we should consider those 2 items as requirements of the use cases. > Or at least consider them as such... before running too far into any model that may break if critical requirements are missed. > > The 2 I am suggesting here - > > 1) ISP's need a MA that can Stand by itself as an Overlay, but more importantly they may need one that operates more globally as a universal MA. > 2) If we adopt a registry and we know virtualization is on it's way, then we need to take that into account (a virtual MA). > I am still trying to understand what you are suggesting. - What does "ISP's need a MA that can Stand by itself as an Overlay" mean? - What does "ISP's ... need one that operates more globally as a universal MA" mean? What is a universal MA? What is more globally here? - What registry are you talking about? It seems you are not talking about the IPPM metrics registry. So what do you have in mind? Reading again your other email, if you are talking about an MA under control of multiple controllers, then I am afraid LMAP 1.0 won't address this (see the WG's charter). But MAs can of course implement tests for multiple metrics. /js -- Juergen Schoenwaelder Jacobs University Bremen gGmbH Phone: +49 421 200 3587 Campus Ring 1, 28759 Bremen, Germany Fax: +49 421 200 3103 From trevor.burbridge@bt.com Tue Nov 12 01:00:28 2013 Return-Path: X-Original-To: lmap@ietfa.amsl.com Delivered-To: lmap@ietfa.amsl.com Received: from localhost (localhost [127.0.0.1]) by ietfa.amsl.com (Postfix) with ESMTP id 734CD21E80DA for ; Tue, 12 Nov 2013 01:00:28 -0800 (PST) X-Virus-Scanned: amavisd-new at amsl.com X-Spam-Flag: NO X-Spam-Score: -3.149 X-Spam-Level: X-Spam-Status: No, score=-3.149 tagged_above=-999 required=5 tests=[AWL=0.451, BAYES_00=-2.599, RCVD_IN_DNSWL_LOW=-1] Received: from mail.ietf.org ([12.22.58.30]) by localhost (ietfa.amsl.com [127.0.0.1]) (amavisd-new, port 10024) with ESMTP id X324sDdFzxbW for ; Tue, 12 Nov 2013 01:00:24 -0800 (PST) Received: from smtpe1.intersmtp.com (smtp64.intersmtp.com [62.239.224.237]) by ietfa.amsl.com (Postfix) with ESMTP id 9D21121E80B0 for ; Tue, 12 Nov 2013 01:00:21 -0800 (PST) Received: from EVMHT65-UKRD.domain1.systemhost.net (10.36.3.102) by RDW083A008ED64.smtp-e4.hygiene.service (10.187.98.13) with Microsoft SMTP Server (TLS) id 8.3.298.1; Tue, 12 Nov 2013 09:00:16 +0000 Received: from EMV64-UKRD.domain1.systemhost.net ([169.254.2.72]) by EVMHT65-UKRD.domain1.systemhost.net ([10.36.3.102]) with mapi; Tue, 12 Nov 2013 09:00:16 +0000 From: To: , Date: Tue, 12 Nov 2013 09:00:15 +0000 Thread-Topic: [lmap] Splitting measurement tasks into measurement tasks plus reporting tasks? Thread-Index: Ac7fHpkl1w8ywBTrQluiflLlHrmNzwAZiaJw Message-ID: References: <20131111182839.GA21454@idrathernotsay.com> <20131111204218.GB770@elstar.local> In-Reply-To: <20131111204218.GB770@elstar.local> Accept-Language: en-US, en-GB Content-Language: en-US X-MS-Has-Attach: X-MS-TNEF-Correlator: acceptlanguage: en-US, en-GB Content-Type: text/plain; charset="us-ascii" Content-Transfer-Encoding: quoted-printable MIME-Version: 1.0 Cc: lmap@ietf.org Subject: Re: [lmap] Splitting measurement tasks into measurement tasks plus reporting tasks? X-BeenThere: lmap@ietf.org X-Mailman-Version: 2.1.12 Precedence: list List-Id: Large Scale Measurement of Access network Performance List-Unsubscribe: , List-Archive: List-Post: List-Help: List-Subscribe: , X-List-Received-Date: Tue, 12 Nov 2013 09:00:28 -0000 What we are discussing is already explicitly supported in exactly the way S= teve has described. The Measurement Task has a schedule and feeds results i= nto one or more reporting channels. These Report Channels then have their o= wn configuration including a schedule in which to report. The intention is that these report channels can also be used for logging/st= atus/alert type information as well as measurement results. Trevor. >-----Original Message----- >From: lmap-bounces@ietf.org [mailto:lmap-bounces@ietf.org] On Behalf Of >Juergen Schoenwaelder >Sent: 11 November 2013 20:42 >To: Steve Miller >Cc: lmap@ietf.org >Subject: Re: [lmap] Splitting measurement tasks into measurement tasks plu= s >reporting tasks? > >On Mon, Nov 11, 2013 at 01:28:39PM -0500, Steve Miller wrote: >> Hi. I was thinking through some of the LMAP model, and to some exten= t >doing so in the context of the (not entirely dissimilar) system we use wit= hin >Verisign. One thing that our internal system includes is some support for= alert >checking and generation, as a thing that can be separate from "pure" >performance reporting. I think that might be a useful capability to have = in LMAP >as well. >> >> In the framework, I'd think we'd represent that by breaking measureme= nt >tasks into two pieces. The first piece would define just the actual measu= rement >task; the second piece would be a separate reporting task, which would con= sume >the results of the measurement task. It'd be possible to feed the results= of a >measurement task into more than one reporting task. >> >> Let's look at the example from page 7 of http://www.ietf.org/id/draft= - >burbridge-lmap-information-model-01.txt: >> >> Example: A Measurement Task Configuration may configure a single >> Measurement Task for measuring UDP latency. The Measurement Task >> Configuration could define the destination port and address for >> the measurement as well as the duration, internal packet timing >> strategy and other parameters (for example a stream for one hour >> and sending one packet every 500 ms). It may also define the >> output type and possible parameters (for example the output type >> can be the 95th percentile mean) where the measurement task >> accepts such parameters. It does NOT define when the task starts >> (this is defined by the Measurement Schedule element), so it does >> not by itself instruct the MA to actually perform this measurement >> task. >> >> Instead of this single measurement task, we'd have the "core" measuremen= t >task: >> >> Perform the UDP latency check against X every 500ms, between the >hours of 10PM and 6AM; call this check "UDP-latency-X" >> >> and then the reporting task implied above: >> >> Consume the output from measurement task "UDP-latency-X", reporting >the 95th percentile mean once an hour; send that report to collector C1 >> >> and then we could also add a second reporting task: >> >> Consume the output from measurement task "UDP-latency-X", >computing the exponentially-decaying average loss over the last 30 seconds= . If >that loss exceeds 5%, report that immediately; do so to collector C2. If = the >exponentially-decaying average loss over the last 30 seconds was in excess= of 5% >but now is less than 3%, report that immediately as well; do so to collect= or C2. >> >> Admittedly, for the regulator use case, I don't know how useful that = is, though >it seems that there would be some value in having devices that were about = to >note a SLA violation not wait for an extended period of time to inform the= carrier >that they are seeing issues. I think that'd help a lot for the separate-b= ut- >overlapping use case I'd articulated previously, in which operators of hea= vily >anycasted infrastructure are looking for devices located in a large number= of ISPs >that they can tell to check their view of the perfomance of those services= , so that >they can know that some corner of the Internet is having issues reaching t= hem. >> >> I don't think this'd be too complicated to implement, either. >> >> Would this be worth considering? If so, I'd be happy to draft some l= anguage >around this. >> > >I hear two suggestions here: > >a) Give reporting channels their own independent schedule (right now > they are essentially tied to the scheduled task). > >b) Provide a means for post-processing raw data in order to derive > certain statistics before the result is shipped via a report > channel. > >Is this a fair summary of the request? > >/js > >-- >Juergen Schoenwaelder Jacobs University Bremen gGmbH >Phone: +49 421 200 3587 Campus Ring 1, 28759 Bremen, Germany >Fax: +49 421 200 3103 >_______________________________________________ >lmap mailing list >lmap@ietf.org >https://www.ietf.org/mailman/listinfo/lmap From philip.eardley@bt.com Tue Nov 12 01:11:56 2013 Return-Path: X-Original-To: lmap@ietfa.amsl.com Delivered-To: lmap@ietfa.amsl.com Received: from localhost (localhost [127.0.0.1]) by ietfa.amsl.com (Postfix) with ESMTP id 12DCF11E80ED for ; Tue, 12 Nov 2013 01:11:56 -0800 (PST) X-Virus-Scanned: amavisd-new at amsl.com X-Spam-Flag: NO X-Spam-Score: -102.753 X-Spam-Level: X-Spam-Status: No, score=-102.753 tagged_above=-999 required=5 tests=[AWL=-0.354, BAYES_00=-2.599, J_CHICKENPOX_72=0.6, J_CHICKENPOX_91=0.6, RCVD_IN_DNSWL_LOW=-1, USER_IN_WHITELIST=-100] Received: from mail.ietf.org ([12.22.58.30]) by localhost (ietfa.amsl.com [127.0.0.1]) (amavisd-new, port 10024) with ESMTP id qyIpcVJDz3OG for ; Tue, 12 Nov 2013 01:11:51 -0800 (PST) Received: from smtpe1.intersmtp.com (smtp64.intersmtp.com [62.239.224.237]) by ietfa.amsl.com (Postfix) with ESMTP id 076E721E80DB for ; Tue, 12 Nov 2013 01:11:42 -0800 (PST) Received: from EVMHT69-UKRD.domain1.systemhost.net (10.36.3.129) by RDW083A008ED64.smtp-e4.hygiene.service (10.187.98.13) with Microsoft SMTP Server (TLS) id 8.3.298.1; Tue, 12 Nov 2013 09:11:40 +0000 Received: from EMV67-UKRD.domain1.systemhost.net ([169.254.2.147]) by EVMHT69-UKRD.domain1.systemhost.net ([10.36.3.129]) with mapi; Tue, 12 Nov 2013 09:11:40 +0000 From: To: , , , , Date: Tue, 12 Nov 2013 09:11:39 +0000 Thread-Topic: Comments & opinions on Draft - RE: [lmap] call for consensus - adoption of draft-burbridge-lmap-information-model as WG document Thread-Index: Ac7bVGevh+4pI7FlSz+236qyIybWMQDqGOVAACJc1qA= Message-ID: References: <9904FB1B0159DA42B0B887B7FA8119CA1292EAA3@AZ-FFEXMB04.global.avaya.com> In-Reply-To: Accept-Language: en-US, en-GB Content-Language: en-US X-MS-Has-Attach: X-MS-TNEF-Correlator: acceptlanguage: en-US, en-GB Content-Type: text/plain; charset="us-ascii" Content-Transfer-Encoding: quoted-printable MIME-Version: 1.0 Cc: Barbara.Stark@bellsouth.com, Charles.Cook2@centurylink.com, acmorton@att.com, KEN.KO@adtran.com Subject: Re: [lmap] Comments & opinions on Draft - RE: call for consensus - adoption of draft-burbridge-lmap-information-model as WG document X-BeenThere: lmap@ietf.org X-Mailman-Version: 2.1.12 Precedence: list List-Id: Large Scale Measurement of Access network Performance List-Unsubscribe: , List-Archive: List-Post: List-Help: List-Subscribe: , X-List-Received-Date: Tue, 12 Nov 2013 09:11:56 -0000 Michael Not sure I fully get your comments, but I think at least some of these issu= es are covered by the framework document. a) MA can do upload and download style tests b) security clearly important c) Instruction can include multiple tests, however only one Controller per = MA (S4.2) d) the start of a test may include a resource check (S5.3) e) direct MA to MA coordination out of scope (S5.5 #1) I haven't really thought about the impact of virtualisation on lmap, but I = agree with your comments about security Best wishes phil > -----Original Message----- > From: Bugenhagen, Michael K > [mailto:Michael.K.Bugenhagen@centurylink.com] > Sent: 11 November 2013 17:05 > To: 'Romascanu, Dan (Dan)'; lmap@ietf.org; Burbridge,T,Trevor,TUB8 R; > Eardley,PL,Philip,TUB8 R; 'marcelo bagnulo braun' > Cc: 'MORTON, ALFRED C (AL)'; Cook, Charles; 'KEN KO'; 'Stark, Barbara' > Subject: Comments & opinions on Draft - RE: [lmap] call for consensus - > adoption of draft-burbridge-lmap-information-model as WG document >=20 > Dan, >=20 > Comment 1. Is written from an ISP Operational team perspective (Scope > says ISP use case so... ) Comment 2. Is future proofing the Registry > concept but it also is in the draft a bit.. >=20 >=20 > 1) ISP use case requirements - (The yet another test system overlay > issue) >=20 > ISP's have multiple IP based "testing systems" / "Measurement > Agents" deployed today, many of which don't co-exist well. So one > ongoing fairly solid ISP requirement that the groups should strongly > consider is the ability of a MA to be open flexible framework MA that > can run both Op's ADHOC test, and performance testing without two MA's. > Example (a MA / probe that can run VoIP, VIDEO, LMAP, DNS testing, > ..... vs. multiple probes doing one each.) >=20 > ** Current draft that means adding the ability to >=20 > ADHOC / Multi-user op's needs for considerations > a) MA capability to both Push and Pull test results > b) MA Radius / Authentication to allow item a) > c) Multiple test capability (OAM, and throughput at the same time by > different Op's teams) > d) Resource availability checking (when you can run more than one test > at a time) > e) MA to MA test capability negotiation (if you are working with a > customers or 3rd party probe) >=20 >=20 >=20 > 2) Future proof the Registry / MA probe concept into Virtual > Applications - (start thinking Virtualization Service Catalog) >=20 > As NFV (network functional virtualization) will likely become > available by the time some of this is being implemented we should > consider adding a "hypervisor" type to the Registry. The thought here > is that Virtual applications come in a default deployable set, *.OVF > files for VMware, ... for KVM, and so on. ** ONE KEY to service > catalogs is a trusted source for the code, one should NEVER (NEVER - > EVER) deploy a virtual appliance from a public source on their > network... >=20 > I think the registry could be a good solution to this issue - provided > we place a high level or safety checks on anything getting loaded into > the registry itself. >=20 >=20 >=20 >=20 >=20 >=20 >=20 >=20 >=20 >=20 >=20 >=20 >=20 >=20 >=20 >=20 >=20 >=20 >=20 >=20 >=20 >=20 >=20 >=20 >=20 >=20 >=20 >=20 >=20 >=20 >=20 >=20 >=20 > -----Original Message----- > From: lmap-bounces@ietf.org [mailto:lmap-bounces@ietf.org] On Behalf Of > Romascanu, Dan (Dan) > Sent: Wednesday, November 06, 2013 6:58 PM > To: lmap@ietf.org > Subject: [lmap] call for consensus - adoption of draft-burbridge-lmap- > information-model as WG document >=20 >=20 > Today in the LMAP meeting we ran a hum on the question whether we > should adopt draft-burbridge-lmap-information-model. The result was > read as showing consensus in favor of the adoption, although there was > at least one opposed opinion expressed by Benoit (as contributor). >=20 > We would like to extend this call to the participants who could not > attend the meeting. Please send your opinions on this question to the > WG mail list before Friday 11/15. >=20 > Thanks and Regards, >=20 > Dan >=20 >=20 >=20 > _______________________________________________ > lmap mailing list > lmap@ietf.org > https://www.ietf.org/mailman/listinfo/lmap From j.schoenwaelder@jacobs-university.de Tue Nov 12 01:17:51 2013 Return-Path: X-Original-To: lmap@ietfa.amsl.com Delivered-To: lmap@ietfa.amsl.com Received: from localhost (localhost [127.0.0.1]) by ietfa.amsl.com (Postfix) with ESMTP id CDF9E21E80C2 for ; Tue, 12 Nov 2013 01:17:50 -0800 (PST) X-Virus-Scanned: amavisd-new at amsl.com X-Spam-Flag: NO X-Spam-Score: -103.199 X-Spam-Level: X-Spam-Status: No, score=-103.199 tagged_above=-999 required=5 tests=[AWL=0.050, BAYES_00=-2.599, HELO_EQ_DE=0.35, RCVD_IN_DNSWL_LOW=-1, USER_IN_WHITELIST=-100] Received: from mail.ietf.org ([12.22.58.30]) by localhost (ietfa.amsl.com [127.0.0.1]) (amavisd-new, port 10024) with ESMTP id MR2zGyDdMGwR for ; Tue, 12 Nov 2013 01:17:44 -0800 (PST) Received: from hermes.jacobs-university.de (hermes.jacobs-university.de [212.201.44.23]) by ietfa.amsl.com (Postfix) with ESMTP id 09A8021E80AC for ; Tue, 12 Nov 2013 01:17:44 -0800 (PST) Received: from localhost (demetrius2.jacobs-university.de [212.201.44.47]) by hermes.jacobs-university.de (Postfix) with ESMTP id 667B120055; Tue, 12 Nov 2013 10:17:40 +0100 (CET) X-Virus-Scanned: amavisd-new at jacobs-university.de Received: from hermes.jacobs-university.de ([212.201.44.23]) by localhost (demetrius2.jacobs-university.de [212.201.44.32]) (amavisd-new, port 10024) with ESMTP id 7glTkp4f4AhQ; Tue, 12 Nov 2013 10:17:40 +0100 (CET) Received: from elstar.local (elstar.jacobs.jacobs-university.de [10.50.231.133]) by hermes.jacobs-university.de (Postfix) with ESMTP id BAA5220039; Tue, 12 Nov 2013 10:17:39 +0100 (CET) Received: by elstar.local (Postfix, from userid 501) id 1AA6C293A903; Tue, 12 Nov 2013 10:17:33 +0100 (CET) Date: Tue, 12 Nov 2013 10:17:33 +0100 From: Juergen Schoenwaelder To: trevor.burbridge@bt.com Message-ID: <20131112091731.GA2503@elstar.local> Mail-Followup-To: trevor.burbridge@bt.com, steve@idrathernotsay.com, lmap@ietf.org References: <20131111182839.GA21454@idrathernotsay.com> <20131111204218.GB770@elstar.local> MIME-Version: 1.0 Content-Type: text/plain; charset=us-ascii Content-Disposition: inline In-Reply-To: User-Agent: Mutt/1.5.21 (2010-09-15) Cc: steve@idrathernotsay.com, lmap@ietf.org Subject: Re: [lmap] Splitting measurement tasks into measurement tasks plus reporting tasks? X-BeenThere: lmap@ietf.org X-Mailman-Version: 2.1.12 Precedence: list Reply-To: Juergen Schoenwaelder List-Id: Large Scale Measurement of Access network Performance List-Unsubscribe: , List-Archive: List-Post: List-Help: List-Subscribe: , X-List-Received-Date: Tue, 12 Nov 2013 09:17:51 -0000 On Tue, Nov 12, 2013 at 09:00:15AM +0000, trevor.burbridge@bt.com wrote: > > What we are discussing is already explicitly supported in exactly the way Steve has described. The Measurement Task has a schedule and feeds results into one or more reporting channels. These Report Channels then have their own configuration including a schedule in which to report. > > The intention is that these report channels can also be used for logging/status/alert type information as well as measurement results. > OK. I stand corrected on a) since the channels include a scheduling mechanism. Concerning status and logging, this is less clear to me since there are now also separate channels for logging and status. According to the framework, it seems the status information goes to the controller not to the collector. The same seems to hold true for logging information. Actually, I think the framework should be clearer. Right now, it says: Control Protocol: The protocol delivering Instruction(s) from a Controller to a Measurement Agent. It also delivers logging information and capabilities information from the Measurement Agent to the Controller. This is pretty clear. But it is silent about status information (or is capability information a subset of status information and hence it is implicit that status information is included). Section 8.4.1 seems to indicate that status is flowing from the MA to the Controller (and I note that logging information does not seem to show up in the protocol model). If the intention is that status and logging information goes to the controller, then perhaps change this to: Control Protocol: The protocol delivering Instruction(s) from a Controller to a Measurement Agent. It also delivers logging information, capability information and status information from the Measurement Agent to the Controller. Do we need to define the terms logging information, capability information and status information? /js -- Juergen Schoenwaelder Jacobs University Bremen gGmbH Phone: +49 421 200 3587 Campus Ring 1, 28759 Bremen, Germany Fax: +49 421 200 3103 From trevor.burbridge@bt.com Tue Nov 12 02:16:11 2013 Return-Path: X-Original-To: lmap@ietfa.amsl.com Delivered-To: lmap@ietfa.amsl.com Received: from localhost (localhost [127.0.0.1]) by ietfa.amsl.com (Postfix) with ESMTP id 63FFD21E80E9 for ; Tue, 12 Nov 2013 02:16:11 -0800 (PST) X-Virus-Scanned: amavisd-new at amsl.com X-Spam-Flag: NO X-Spam-Score: -2.913 X-Spam-Level: X-Spam-Status: No, score=-2.913 tagged_above=-999 required=5 tests=[AWL=0.086, BAYES_00=-2.599, J_CHICKENPOX_91=0.6, RCVD_IN_DNSWL_LOW=-1] Received: from mail.ietf.org ([12.22.58.30]) by localhost (ietfa.amsl.com [127.0.0.1]) (amavisd-new, port 10024) with ESMTP id Y8oPNm1zP6F2 for ; Tue, 12 Nov 2013 02:15:59 -0800 (PST) Received: from smtpe1.intersmtp.com (smtp63.intersmtp.com [62.239.224.236]) by ietfa.amsl.com (Postfix) with ESMTP id AF39E11E80F1 for ; Tue, 12 Nov 2013 02:15:58 -0800 (PST) Received: from EVMHT64-UKRD.domain1.systemhost.net (10.36.3.101) by RDW083A007ED63.smtp-e3.hygiene.service (10.187.98.12) with Microsoft SMTP Server (TLS) id 8.3.298.1; Tue, 12 Nov 2013 10:15:57 +0000 Received: from EMV64-UKRD.domain1.systemhost.net ([169.254.2.72]) by EVMHT64-UKRD.domain1.systemhost.net ([10.36.3.101]) with mapi; Tue, 12 Nov 2013 10:15:57 +0000 From: To: Date: Tue, 12 Nov 2013 10:15:56 +0000 Thread-Topic: [lmap] Splitting measurement tasks into measurement tasks plus reporting tasks? Thread-Index: Ac7fiAs17PanNrNNSZqJ2tNjxKhRvgAB9ijQ Message-ID: References: <20131111182839.GA21454@idrathernotsay.com> <20131111204218.GB770@elstar.local> <20131112091731.GA2503@elstar.local> In-Reply-To: <20131112091731.GA2503@elstar.local> Accept-Language: en-US, en-GB Content-Language: en-US X-MS-Has-Attach: X-MS-TNEF-Correlator: acceptlanguage: en-US, en-GB Content-Type: text/plain; charset="us-ascii" Content-Transfer-Encoding: quoted-printable MIME-Version: 1.0 Cc: steve@idrathernotsay.com, lmap@ietf.org Subject: Re: [lmap] Splitting measurement tasks into measurement tasks plus reporting tasks? X-BeenThere: lmap@ietf.org X-Mailman-Version: 2.1.12 Precedence: list List-Id: Large Scale Measurement of Access network Performance List-Unsubscribe: , List-Archive: List-Post: List-Help: List-Subscribe: , X-List-Received-Date: Tue, 12 Nov 2013 10:16:11 -0000 Yes we probably do need to clear this up in the next iteration of the infor= mation model and architecture documents. A Report Channel in the information model can be set with a URL of a Contro= ller interface to report logging and status information. However, we probab= ly should rename it. Trevor. >-----Original Message----- >From: Juergen Schoenwaelder [mailto:j.schoenwaelder@jacobs-university.de] >Sent: 12 November 2013 09:18 >To: Burbridge,T,Trevor,TUB8 R >Cc: steve@idrathernotsay.com; lmap@ietf.org >Subject: Re: [lmap] Splitting measurement tasks into measurement tasks plu= s >reporting tasks? > >On Tue, Nov 12, 2013 at 09:00:15AM +0000, trevor.burbridge@bt.com wrote: >> >> What we are discussing is already explicitly supported in exactly the wa= y Steve >has described. The Measurement Task has a schedule and feeds results into = one >or more reporting channels. These Report Channels then have their own >configuration including a schedule in which to report. >> >> The intention is that these report channels can also be used for >logging/status/alert type information as well as measurement results. >> > >OK. I stand corrected on a) since the channels include a scheduling mechan= ism. >Concerning status and logging, this is less clear to me since there are no= w also >separate channels for logging and status. >According to the framework, it seems the status information goes to the >controller not to the collector. The same seems to hold true for logging >information. Actually, I think the framework should be clearer. Right now,= it says: > > Control Protocol: The protocol delivering Instruction(s) from a > Controller to a Measurement Agent. It also delivers logging > information and capabilities information from the Measurement Agent > to the Controller. > >This is pretty clear. But it is silent about status information (or is cap= ability >information a subset of status information and hence it is implicit that s= tatus >information is included). Section 8.4.1 seems to indicate that status is f= lowing >from the MA to the Controller (and I note that logging information does no= t seem >to show up in the protocol model). If the intention is that status and log= ging >information goes to the controller, then perhaps change this to: > > Control Protocol: The protocol delivering Instruction(s) from a > Controller to a Measurement Agent. It also delivers logging > information, capability information and status information from the > Measurement Agent to the Controller. > >Do we need to define the terms logging information, capability information= and >status information? > >/js > >-- >Juergen Schoenwaelder Jacobs University Bremen gGmbH >Phone: +49 421 200 3587 Campus Ring 1, 28759 Bremen, Germany >Fax: +49 421 200 3103 From dromasca@avaya.com Tue Nov 12 02:39:12 2013 Return-Path: X-Original-To: lmap@ietfa.amsl.com Delivered-To: lmap@ietfa.amsl.com Received: from localhost (localhost [127.0.0.1]) by ietfa.amsl.com (Postfix) with ESMTP id 27F7921E80C1 for ; Tue, 12 Nov 2013 02:39:12 -0800 (PST) X-Virus-Scanned: amavisd-new at amsl.com X-Spam-Flag: NO X-Spam-Score: -103.332 X-Spam-Level: X-Spam-Status: No, score=-103.332 tagged_above=-999 required=5 tests=[AWL=0.267, BAYES_00=-2.599, RCVD_IN_DNSWL_LOW=-1, USER_IN_WHITELIST=-100] Received: from mail.ietf.org ([12.22.58.30]) by localhost (ietfa.amsl.com [127.0.0.1]) (amavisd-new, port 10024) with ESMTP id 4+Q1Am78sDkl for ; Tue, 12 Nov 2013 02:39:06 -0800 (PST) Received: from de307622-de-outbound.net.avaya.com (de307622-de-outbound.net.avaya.com [198.152.71.100]) by ietfa.amsl.com (Postfix) with ESMTP id A86F111E80EE for ; Tue, 12 Nov 2013 02:39:05 -0800 (PST) X-IronPort-Anti-Spam-Filtered: true X-IronPort-Anti-Spam-Result: AjwFACkEglKHCzI1/2dsb2JhbABZgmYhOFO/HIErFnSCJQEBAQEDAQEBDwsdNAsMBAIBCA0EBAEBCxQFBAcnCxQJCAEBBAENBQgTB4dfAQyhTZxhEwSOJoEIMQcGgxqBEQOec4NvhzeBaIE+gWgHAhci X-IronPort-AV: E=Sophos;i="4.93,684,1378872000"; d="scan'208";a="31278611" Received: from unknown (HELO p-us1-erheast.us1.avaya.com) ([135.11.50.53]) by de307622-de-outbound.net.avaya.com with ESMTP; 12 Nov 2013 05:38:53 -0500 Received: from unknown (HELO AZ-FFEXHC04.global.avaya.com) ([135.64.58.14]) by p-us1-erheast-out.us1.avaya.com with ESMTP; 12 Nov 2013 05:28:43 -0500 Received: from AZ-FFEXMB04.global.avaya.com ([fe80::6db7:b0af:8480:c126]) by AZ-FFEXHC04.global.avaya.com ([135.64.58.14]) with mapi id 14.03.0146.000; Tue, 12 Nov 2013 11:38:50 +0100 From: "Romascanu, Dan (Dan)" To: "Bugenhagen, Michael K" , "lmap@ietf.org" , 'Trevor Burbridge' , "'philip.eardley@bt.com'" , 'marcelo bagnulo braun' Thread-Topic: Comments & opinions on Draft - RE: [lmap] call for consensus - adoption of draft-burbridge-lmap-information-model as WG document Thread-Index: Ac7bVGevh+4pI7FlSz+236qyIybWMQDqGOVAACVfjoA= Date: Tue, 12 Nov 2013 10:38:50 +0000 Message-ID: <9904FB1B0159DA42B0B887B7FA8119CA1293A32C@AZ-FFEXMB04.global.avaya.com> References: <9904FB1B0159DA42B0B887B7FA8119CA1292EAA3@AZ-FFEXMB04.global.avaya.com> In-Reply-To: Accept-Language: en-US Content-Language: en-US X-MS-Has-Attach: X-MS-TNEF-Correlator: x-originating-ip: [135.64.58.46] Content-Type: text/plain; charset="us-ascii" Content-Transfer-Encoding: quoted-printable MIME-Version: 1.0 Cc: "'Stark, Barbara'" , "Cook, Charles" , "'MORTON, ALFRED C \(AL\)'" , 'KEN KO' Subject: Re: [lmap] Comments & opinions on Draft - RE: call for consensus - adoption of draft-burbridge-lmap-information-model as WG document X-BeenThere: lmap@ietf.org X-Mailman-Version: 2.1.12 Precedence: list List-Id: Large Scale Measurement of Access network Performance List-Unsubscribe: , List-Archive: List-Post: List-Help: List-Subscribe: , X-List-Received-Date: Tue, 12 Nov 2013 10:39:12 -0000 Hi Michael, Thank you for your comments. As I can see the authors started to address th= em.=20 My question (wearing the WG chair hat :-) ) - are you OK with the adoption = of draft-burbridge-lmap-information-model as WG document? Saying 'no' means= that you consider that the document needs quite serious changes before bec= oming a WG document, maybe that you have or you are aware about one ore mor= e alternate proposals to be submitted. Saying 'yes' means the document is o= n the right direction, with changes and improvements that can be made in th= e following iterations.=20 Thanks and Regards, Dan > -----Original Message----- > From: Bugenhagen, Michael K > [mailto:Michael.K.Bugenhagen@centurylink.com] > Sent: Monday, November 11, 2013 7:05 PM > To: Romascanu, Dan (Dan); lmap@ietf.org; 'Trevor Burbridge'; > 'philip.eardley@bt.com'; 'marcelo bagnulo braun' > Cc: 'MORTON, ALFRED C (AL)'; Cook, Charles; 'KEN KO'; 'Stark, Barbara' > Subject: Comments & opinions on Draft - RE: [lmap] call for consensus - > adoption of draft-burbridge-lmap-information-model as WG document >=20 > Dan, >=20 > Comment 1. Is written from an ISP Operational team perspective (Scope > says ISP use case so... ) Comment 2. Is future proofing the Registry > concept but it also is in the draft a bit.. >=20 >=20 > 1) ISP use case requirements - (The yet another test system overlay > issue) >=20 > ISP's have multiple IP based "testing systems" / > "Measurement Agents" deployed today, many of which don't co-exist well. > So one ongoing fairly solid ISP requirement that the groups should > strongly consider is the ability of a MA to be open flexible framework > MA that can run both Op's ADHOC test, and performance testing without > two MA's. Example (a MA / probe that can run VoIP, VIDEO, LMAP, DNS > testing, ..... vs. multiple probes doing one each.) >=20 > ** Current draft that means adding the ability to >=20 > ADHOC / Multi-user op's needs for considerations > a) MA capability to both Push and Pull test results > b) MA Radius / Authentication to allow item a) > c) Multiple test capability (OAM, and throughput at the same time by > different Op's teams) > d) Resource availability checking (when you can run more than one test > at a time) > e) MA to MA test capability negotiation (if you are working with a > customers or 3rd party probe) >=20 >=20 >=20 > 2) Future proof the Registry / MA probe concept into Virtual > Applications - (start thinking Virtualization Service Catalog) >=20 > As NFV (network functional virtualization) will likely become > available by the time some of this is being implemented we should > consider adding a "hypervisor" type to the Registry. The thought here > is that Virtual applications come in a default deployable set, *.OVF > files for VMware, ... for KVM, and so on. ** ONE KEY to service > catalogs is a trusted source for the code, one should NEVER (NEVER - > EVER) deploy a virtual appliance from a public source on their > network... >=20 > I think the registry could be a good solution to this issue - provided > we place a high level or safety checks on anything getting loaded into > the registry itself. >=20 >=20 >=20 >=20 >=20 >=20 >=20 >=20 >=20 >=20 >=20 >=20 >=20 >=20 >=20 >=20 >=20 >=20 >=20 >=20 >=20 >=20 >=20 >=20 >=20 >=20 >=20 >=20 >=20 >=20 >=20 >=20 >=20 > -----Original Message----- > From: lmap-bounces@ietf.org [mailto:lmap-bounces@ietf.org] On Behalf Of > Romascanu, Dan (Dan) > Sent: Wednesday, November 06, 2013 6:58 PM > To: lmap@ietf.org > Subject: [lmap] call for consensus - adoption of draft-burbridge-lmap- > information-model as WG document >=20 >=20 > Today in the LMAP meeting we ran a hum on the question whether we should > adopt draft-burbridge-lmap-information-model. The result was read as > showing consensus in favor of the adoption, although there was at least > one opposed opinion expressed by Benoit (as contributor). >=20 > We would like to extend this call to the participants who could not > attend the meeting. Please send your opinions on this question to the WG > mail list before Friday 11/15. >=20 > Thanks and Regards, >=20 > Dan >=20 >=20 >=20 > _______________________________________________ > lmap mailing list > lmap@ietf.org > https://www.ietf.org/mailman/listinfo/lmap From j.schoenwaelder@jacobs-university.de Tue Nov 12 02:53:06 2013 Return-Path: X-Original-To: lmap@ietfa.amsl.com Delivered-To: lmap@ietfa.amsl.com Received: from localhost (localhost [127.0.0.1]) by ietfa.amsl.com (Postfix) with ESMTP id B384611E812F for ; Tue, 12 Nov 2013 02:53:06 -0800 (PST) X-Virus-Scanned: amavisd-new at amsl.com X-Spam-Flag: NO X-Spam-Score: -103.2 X-Spam-Level: X-Spam-Status: No, score=-103.2 tagged_above=-999 required=5 tests=[AWL=0.049, BAYES_00=-2.599, HELO_EQ_DE=0.35, RCVD_IN_DNSWL_LOW=-1, USER_IN_WHITELIST=-100] Received: from mail.ietf.org ([12.22.58.30]) by localhost (ietfa.amsl.com [127.0.0.1]) (amavisd-new, port 10024) with ESMTP id y3NCMsBlIXSs for ; Tue, 12 Nov 2013 02:52:58 -0800 (PST) Received: from hermes.jacobs-university.de (hermes.jacobs-university.de [212.201.44.23]) by ietfa.amsl.com (Postfix) with ESMTP id 333F711E812B for ; Tue, 12 Nov 2013 02:52:58 -0800 (PST) Received: from localhost (demetrius3.jacobs-university.de [212.201.44.48]) by hermes.jacobs-university.de (Postfix) with ESMTP id 94CE62006F; Tue, 12 Nov 2013 11:52:57 +0100 (CET) X-Virus-Scanned: amavisd-new at jacobs-university.de Received: from hermes.jacobs-university.de ([212.201.44.23]) by localhost (demetrius3.jacobs-university.de [212.201.44.32]) (amavisd-new, port 10024) with ESMTP id adIt55H09HBG; Tue, 12 Nov 2013 11:52:57 +0100 (CET) Received: from elstar.local (elstar.jacobs.jacobs-university.de [10.50.231.133]) by hermes.jacobs-university.de (Postfix) with ESMTP id 259072006D; Tue, 12 Nov 2013 11:52:57 +0100 (CET) Received: by elstar.local (Postfix, from userid 501) id 7F103293AFA6; Tue, 12 Nov 2013 11:52:52 +0100 (CET) Date: Tue, 12 Nov 2013 11:52:52 +0100 From: Juergen Schoenwaelder To: trevor.burbridge@bt.com Message-ID: <20131112105252.GB2891@elstar.local> Mail-Followup-To: trevor.burbridge@bt.com, steve@idrathernotsay.com, lmap@ietf.org References: <20131111182839.GA21454@idrathernotsay.com> <20131111204218.GB770@elstar.local> <20131112091731.GA2503@elstar.local> MIME-Version: 1.0 Content-Type: text/plain; charset=us-ascii Content-Disposition: inline In-Reply-To: User-Agent: Mutt/1.5.21 (2010-09-15) Cc: steve@idrathernotsay.com, lmap@ietf.org Subject: Re: [lmap] Splitting measurement tasks into measurement tasks plus reporting tasks? X-BeenThere: lmap@ietf.org X-Mailman-Version: 2.1.12 Precedence: list Reply-To: Juergen Schoenwaelder List-Id: Large Scale Measurement of Access network Performance List-Unsubscribe: , List-Archive: List-Post: List-Help: List-Subscribe: , X-List-Received-Date: Tue, 12 Nov 2013 10:53:06 -0000 On Tue, Nov 12, 2013 at 10:15:56AM +0000, trevor.burbridge@bt.com wrote: > > Yes we probably do need to clear this up in the next iteration of the information model and architecture documents. > > A Report Channel in the information model can be set with a URL of a Controller interface to report logging and status information. However, we probably should rename it. > Now I am getting really confused. Do you believe status/logging is going via the Report Channel?? /js -- Juergen Schoenwaelder Jacobs University Bremen gGmbH Phone: +49 421 200 3587 Campus Ring 1, 28759 Bremen, Germany Fax: +49 421 200 3103 From trevor.burbridge@bt.com Tue Nov 12 03:02:13 2013 Return-Path: X-Original-To: lmap@ietfa.amsl.com Delivered-To: lmap@ietfa.amsl.com Received: from localhost (localhost [127.0.0.1]) by ietfa.amsl.com (Postfix) with ESMTP id D822211E80F1 for ; Tue, 12 Nov 2013 03:02:13 -0800 (PST) X-Virus-Scanned: amavisd-new at amsl.com X-Spam-Flag: NO X-Spam-Score: -2.924 X-Spam-Level: X-Spam-Status: No, score=-2.924 tagged_above=-999 required=5 tests=[AWL=0.075, BAYES_00=-2.599, J_CHICKENPOX_91=0.6, RCVD_IN_DNSWL_LOW=-1] Received: from mail.ietf.org ([12.22.58.30]) by localhost (ietfa.amsl.com [127.0.0.1]) (amavisd-new, port 10024) with ESMTP id gkYQ+fJUDc9x for ; Tue, 12 Nov 2013 03:02:09 -0800 (PST) Received: from smtpe1.intersmtp.com (smtp63.intersmtp.com [62.239.224.236]) by ietfa.amsl.com (Postfix) with ESMTP id 26AC911E8125 for ; Tue, 12 Nov 2013 03:02:08 -0800 (PST) Received: from EVMHT61-UKRD.domain1.systemhost.net (10.36.3.127) by RDW083A007ED63.smtp-e3.hygiene.service (10.187.98.12) with Microsoft SMTP Server (TLS) id 8.3.298.1; Tue, 12 Nov 2013 11:02:06 +0000 Received: from EMV64-UKRD.domain1.systemhost.net ([169.254.2.72]) by EVMHT61-UKRD.domain1.systemhost.net ([10.36.3.127]) with mapi; Tue, 12 Nov 2013 11:02:06 +0000 From: To: Date: Tue, 12 Nov 2013 11:02:05 +0000 Thread-Topic: [lmap] Splitting measurement tasks into measurement tasks plus reporting tasks? Thread-Index: Ac7flV4gD28c1M+QTlGxE4Y5j6n9owAAPUGg Message-ID: References: <20131111182839.GA21454@idrathernotsay.com> <20131111204218.GB770@elstar.local> <20131112091731.GA2503@elstar.local> <20131112105252.GB2891@elstar.local> In-Reply-To: <20131112105252.GB2891@elstar.local> Accept-Language: en-US, en-GB Content-Language: en-US X-MS-Has-Attach: X-MS-TNEF-Correlator: acceptlanguage: en-US, en-GB Content-Type: text/plain; charset="us-ascii" Content-Transfer-Encoding: quoted-printable MIME-Version: 1.0 Cc: steve@idrathernotsay.com, lmap@ietf.org Subject: Re: [lmap] Splitting measurement tasks into measurement tasks plus reporting tasks? X-BeenThere: lmap@ietf.org X-Mailman-Version: 2.1.12 Precedence: list List-Id: Large Scale Measurement of Access network Performance List-Unsubscribe: , List-Archive: List-Post: List-Help: List-Subscribe: , X-List-Received-Date: Tue, 12 Nov 2013 11:02:14 -0000 Not _the_ Report Channel, but _a_ Report Channel, yes.=20 We already have multiple channels to split or duplicate measurement results= so there is no problem re-using that part of the information model for the= delivery of other data. Trevor. >-----Original Message----- >From: Juergen Schoenwaelder [mailto:j.schoenwaelder@jacobs-university.de] >Sent: 12 November 2013 10:53 >To: Burbridge,T,Trevor,TUB8 R >Cc: steve@idrathernotsay.com; lmap@ietf.org >Subject: Re: [lmap] Splitting measurement tasks into measurement tasks plu= s >reporting tasks? > >On Tue, Nov 12, 2013 at 10:15:56AM +0000, trevor.burbridge@bt.com wrote: >> >> Yes we probably do need to clear this up in the next iteration of the in= formation >model and architecture documents. >> >> A Report Channel in the information model can be set with a URL of a >Controller interface to report logging and status information. However, we >probably should rename it. >> > >Now I am getting really confused. Do you believe status/logging is going v= ia the >Report Channel?? > >/js > >-- >Juergen Schoenwaelder Jacobs University Bremen gGmbH >Phone: +49 421 200 3587 Campus Ring 1, 28759 Bremen, Germany >Fax: +49 421 200 3103 From j.schoenwaelder@jacobs-university.de Tue Nov 12 03:10:30 2013 Return-Path: X-Original-To: lmap@ietfa.amsl.com Delivered-To: lmap@ietfa.amsl.com Received: from localhost (localhost [127.0.0.1]) by ietfa.amsl.com (Postfix) with ESMTP id 1518611E8127 for ; Tue, 12 Nov 2013 03:10:30 -0800 (PST) X-Virus-Scanned: amavisd-new at amsl.com X-Spam-Flag: NO X-Spam-Score: -103.201 X-Spam-Level: X-Spam-Status: No, score=-103.201 tagged_above=-999 required=5 tests=[AWL=0.048, BAYES_00=-2.599, HELO_EQ_DE=0.35, RCVD_IN_DNSWL_LOW=-1, USER_IN_WHITELIST=-100] Received: from mail.ietf.org ([12.22.58.30]) by localhost (ietfa.amsl.com [127.0.0.1]) (amavisd-new, port 10024) with ESMTP id 8V35v94sm04x for ; Tue, 12 Nov 2013 03:10:16 -0800 (PST) Received: from hermes.jacobs-university.de (hermes.jacobs-university.de [212.201.44.23]) by ietfa.amsl.com (Postfix) with ESMTP id 4451311E80F5 for ; Tue, 12 Nov 2013 03:10:16 -0800 (PST) Received: from localhost (demetrius3.jacobs-university.de [212.201.44.48]) by hermes.jacobs-university.de (Postfix) with ESMTP id A803820056; Tue, 12 Nov 2013 12:10:15 +0100 (CET) X-Virus-Scanned: amavisd-new at jacobs-university.de Received: from hermes.jacobs-university.de ([212.201.44.23]) by localhost (demetrius3.jacobs-university.de [212.201.44.32]) (amavisd-new, port 10024) with ESMTP id x6Z_ImMvUGRv; Tue, 12 Nov 2013 12:10:15 +0100 (CET) Received: from elstar.local (elstar.jacobs.jacobs-university.de [10.50.231.133]) by hermes.jacobs-university.de (Postfix) with ESMTP id 3197020055; Tue, 12 Nov 2013 12:10:15 +0100 (CET) Received: by elstar.local (Postfix, from userid 501) id 91CAC293B11F; Tue, 12 Nov 2013 12:10:10 +0100 (CET) Date: Tue, 12 Nov 2013 12:10:10 +0100 From: Juergen Schoenwaelder To: trevor.burbridge@bt.com Message-ID: <20131112111010.GD2891@elstar.local> Mail-Followup-To: trevor.burbridge@bt.com, steve@idrathernotsay.com, lmap@ietf.org References: <20131111182839.GA21454@idrathernotsay.com> <20131111204218.GB770@elstar.local> <20131112091731.GA2503@elstar.local> <20131112105252.GB2891@elstar.local> MIME-Version: 1.0 Content-Type: text/plain; charset=us-ascii Content-Disposition: inline In-Reply-To: User-Agent: Mutt/1.5.21 (2010-09-15) Cc: steve@idrathernotsay.com, lmap@ietf.org Subject: Re: [lmap] Splitting measurement tasks into measurement tasks plus reporting tasks? X-BeenThere: lmap@ietf.org X-Mailman-Version: 2.1.12 Precedence: list Reply-To: Juergen Schoenwaelder List-Id: Large Scale Measurement of Access network Performance List-Unsubscribe: , List-Archive: List-Post: List-Help: List-Subscribe: , X-List-Received-Date: Tue, 12 Nov 2013 11:10:30 -0000 On Tue, Nov 12, 2013 at 11:02:05AM +0000, trevor.burbridge@bt.com wrote: > > Not _the_ Report Channel, but _a_ Report Channel, yes. > > We already have multiple channels to split or duplicate measurement results so there is no problem re-using that part of the information model for the delivery of other data. > A report channel by definition delivers to a collector. So which information do you think goes where? (I do not care about 'the' or 'a' - I am lost somewhere else.) /js -- Juergen Schoenwaelder Jacobs University Bremen gGmbH Phone: +49 421 200 3587 Campus Ring 1, 28759 Bremen, Germany Fax: +49 421 200 3103 From trevor.burbridge@bt.com Tue Nov 12 03:13:01 2013 Return-Path: X-Original-To: lmap@ietfa.amsl.com Delivered-To: lmap@ietfa.amsl.com Received: from localhost (localhost [127.0.0.1]) by ietfa.amsl.com (Postfix) with ESMTP id 2B7A611E812B for ; Tue, 12 Nov 2013 03:13:01 -0800 (PST) X-Virus-Scanned: amavisd-new at amsl.com X-Spam-Flag: NO X-Spam-Score: -2.932 X-Spam-Level: X-Spam-Status: No, score=-2.932 tagged_above=-999 required=5 tests=[AWL=0.067, BAYES_00=-2.599, J_CHICKENPOX_91=0.6, RCVD_IN_DNSWL_LOW=-1] Received: from mail.ietf.org ([12.22.58.30]) by localhost (ietfa.amsl.com [127.0.0.1]) (amavisd-new, port 10024) with ESMTP id MpqzGHhRFL09 for ; Tue, 12 Nov 2013 03:12:56 -0800 (PST) Received: from smtpe1.intersmtp.com (smtp63.intersmtp.com [62.239.224.236]) by ietfa.amsl.com (Postfix) with ESMTP id 6C80F11E8127 for ; Tue, 12 Nov 2013 03:12:56 -0800 (PST) Received: from EVMHT66-UKRD.domain1.systemhost.net (10.36.3.103) by RDW083A007ED63.smtp-e3.hygiene.service (10.187.98.12) with Microsoft SMTP Server (TLS) id 8.3.298.1; Tue, 12 Nov 2013 11:12:55 +0000 Received: from EMV64-UKRD.domain1.systemhost.net ([169.254.2.72]) by EVMHT66-UKRD.domain1.systemhost.net ([10.36.3.103]) with mapi; Tue, 12 Nov 2013 11:12:55 +0000 From: To: Date: Tue, 12 Nov 2013 11:12:53 +0000 Thread-Topic: [lmap] Splitting measurement tasks into measurement tasks plus reporting tasks? Thread-Index: Ac7fl8h/NlqZRTE2Q8aF6pX4Zb9RFQAAAhMA Message-ID: References: <20131111182839.GA21454@idrathernotsay.com> <20131111204218.GB770@elstar.local> <20131112091731.GA2503@elstar.local> <20131112105252.GB2891@elstar.local> <20131112111010.GD2891@elstar.local> In-Reply-To: <20131112111010.GD2891@elstar.local> Accept-Language: en-US, en-GB Content-Language: en-US X-MS-Has-Attach: X-MS-TNEF-Correlator: acceptlanguage: en-US, en-GB Content-Type: text/plain; charset="us-ascii" Content-Transfer-Encoding: quoted-printable MIME-Version: 1.0 Cc: steve@idrathernotsay.com, lmap@ietf.org Subject: Re: [lmap] Splitting measurement tasks into measurement tasks plus reporting tasks? X-BeenThere: lmap@ietf.org X-Mailman-Version: 2.1.12 Precedence: list List-Id: Large Scale Measurement of Access network Performance List-Unsubscribe: , List-Archive: List-Post: List-Help: List-Subscribe: , X-List-Received-Date: Tue, 12 Nov 2013 11:13:01 -0000 In the information model the report channel delivers information to a speci= fied URL. I don't see why that can't be the Controller. Or if we want we ca= n talk about the Controller having a Collector interface we can do that as = well. This is all terminology. As I said we need to tidy up some of the naming and definitions, but techni= cally I don't see any problems - we designed it to work this way. Trevor. >-----Original Message----- >From: Juergen Schoenwaelder [mailto:j.schoenwaelder@jacobs-university.de] >Sent: 12 November 2013 11:10 >To: Burbridge,T,Trevor,TUB8 R >Cc: steve@idrathernotsay.com; lmap@ietf.org >Subject: Re: [lmap] Splitting measurement tasks into measurement tasks plu= s >reporting tasks? > >On Tue, Nov 12, 2013 at 11:02:05AM +0000, trevor.burbridge@bt.com wrote: >> >> Not _the_ Report Channel, but _a_ Report Channel, yes. >> >> We already have multiple channels to split or duplicate measurement resu= lts so >there is no problem re-using that part of the information model for the de= livery >of other data. >> > >A report channel by definition delivers to a collector. So which informati= on do >you think goes where? (I do not care about 'the' or 'a' >- I am lost somewhere else.) > >/js > >-- >Juergen Schoenwaelder Jacobs University Bremen gGmbH >Phone: +49 421 200 3587 Campus Ring 1, 28759 Bremen, Germany >Fax: +49 421 200 3103 From j.schoenwaelder@jacobs-university.de Tue Nov 12 03:20:31 2013 Return-Path: X-Original-To: lmap@ietfa.amsl.com Delivered-To: lmap@ietfa.amsl.com Received: from localhost (localhost [127.0.0.1]) by ietfa.amsl.com (Postfix) with ESMTP id D773B11E813D for ; Tue, 12 Nov 2013 03:20:31 -0800 (PST) X-Virus-Scanned: amavisd-new at amsl.com X-Spam-Flag: NO X-Spam-Score: -103.202 X-Spam-Level: X-Spam-Status: No, score=-103.202 tagged_above=-999 required=5 tests=[AWL=0.047, BAYES_00=-2.599, HELO_EQ_DE=0.35, RCVD_IN_DNSWL_LOW=-1, USER_IN_WHITELIST=-100] Received: from mail.ietf.org ([12.22.58.30]) by localhost (ietfa.amsl.com [127.0.0.1]) (amavisd-new, port 10024) with ESMTP id 1Lo2eRTm4ebX for ; Tue, 12 Nov 2013 03:20:26 -0800 (PST) Received: from hermes.jacobs-university.de (hermes.jacobs-university.de [212.201.44.23]) by ietfa.amsl.com (Postfix) with ESMTP id 0AB8A11E8125 for ; Tue, 12 Nov 2013 03:20:26 -0800 (PST) Received: from localhost (demetrius1.jacobs-university.de [212.201.44.46]) by hermes.jacobs-university.de (Postfix) with ESMTP id 6E80C2004F; Tue, 12 Nov 2013 12:20:25 +0100 (CET) X-Virus-Scanned: amavisd-new at jacobs-university.de Received: from hermes.jacobs-university.de ([212.201.44.23]) by localhost (demetrius1.jacobs-university.de [212.201.44.32]) (amavisd-new, port 10024) with ESMTP id qFyfWQFEA8OQ; Tue, 12 Nov 2013 12:20:25 +0100 (CET) Received: from elstar.local (elstar.jacobs.jacobs-university.de [10.50.231.133]) by hermes.jacobs-university.de (Postfix) with ESMTP id 005F720042; Tue, 12 Nov 2013 12:20:24 +0100 (CET) Received: by elstar.local (Postfix, from userid 501) id 2A36A293B2D6; Tue, 12 Nov 2013 12:20:20 +0100 (CET) Date: Tue, 12 Nov 2013 12:20:20 +0100 From: Juergen Schoenwaelder To: trevor.burbridge@bt.com Message-ID: <20131112112019.GF2891@elstar.local> Mail-Followup-To: trevor.burbridge@bt.com, steve@idrathernotsay.com, lmap@ietf.org References: <20131111182839.GA21454@idrathernotsay.com> <20131111204218.GB770@elstar.local> <20131112091731.GA2503@elstar.local> <20131112105252.GB2891@elstar.local> <20131112111010.GD2891@elstar.local> MIME-Version: 1.0 Content-Type: text/plain; charset=us-ascii Content-Disposition: inline In-Reply-To: User-Agent: Mutt/1.5.21 (2010-09-15) Cc: steve@idrathernotsay.com, lmap@ietf.org Subject: Re: [lmap] Splitting measurement tasks into measurement tasks plus reporting tasks? X-BeenThere: lmap@ietf.org X-Mailman-Version: 2.1.12 Precedence: list Reply-To: Juergen Schoenwaelder List-Id: Large Scale Measurement of Access network Performance List-Unsubscribe: , List-Archive: List-Post: List-Help: List-Subscribe: , X-List-Received-Date: Tue, 12 Nov 2013 11:20:32 -0000 On Tue, Nov 12, 2013 at 11:12:53AM +0000, trevor.burbridge@bt.com wrote: > > In the information model the report channel delivers information to a specified URL. I don't see why that can't be the Controller. Or if we want we can talk about the Controller having a Collector interface we can do that as well. This is all terminology. > > As I said we need to tidy up some of the naming and definitions, but technically I don't see any problems - we designed it to work this way. > For me, this is not at all clear. In fact, the latest info model also has a status channel and a logging channel. It is great if we can ship stuff anywhere - but I think we would be even better off if we have a clear view who should be the receiver / consumer of status and logging information. I think this is actually an architectural question to answer. /js -- Juergen Schoenwaelder Jacobs University Bremen gGmbH Phone: +49 421 200 3587 Campus Ring 1, 28759 Bremen, Germany Fax: +49 421 200 3103 From ken.ko@adtran.com Tue Nov 12 03:49:33 2013 Return-Path: X-Original-To: lmap@ietfa.amsl.com Delivered-To: lmap@ietfa.amsl.com Received: from localhost (localhost [127.0.0.1]) by ietfa.amsl.com (Postfix) with ESMTP id A54E511E8105 for ; Tue, 12 Nov 2013 03:49:33 -0800 (PST) X-Virus-Scanned: amavisd-new at amsl.com X-Spam-Flag: NO X-Spam-Score: -6.599 X-Spam-Level: X-Spam-Status: No, score=-6.599 tagged_above=-999 required=5 tests=[BAYES_00=-2.599, RCVD_IN_DNSWL_MED=-4] Received: from mail.ietf.org ([12.22.58.30]) by localhost (ietfa.amsl.com [127.0.0.1]) (amavisd-new, port 10024) with ESMTP id pbD566SHxWhp for ; Tue, 12 Nov 2013 03:49:28 -0800 (PST) Received: from p02c11o148.mxlogic.net (p02c11o148.mxlogic.net [208.65.144.81]) by ietfa.amsl.com (Postfix) with ESMTP id B0D2B11E80F8 for ; Tue, 12 Nov 2013 03:49:23 -0800 (PST) Received: from unknown [76.164.174.81] (EHLO ex-hc1.corp.adtran.com) by p02c11o148.mxlogic.net(mxl_mta-7.2.2-0) with ESMTP id 4c512825.2af3b6e0a940.20471.00-593.30291.p02c11o148.mxlogic.net (envelope-from ); Tue, 12 Nov 2013 04:49:24 -0700 (MST) X-MXL-Hash: 528215c41a8570be-f1c15ef2c607f074b704a65313a86b30ad784634 Received: from unknown [76.164.174.81] (EHLO ex-hc1.corp.adtran.com) by p02c11o148.mxlogic.net(mxl_mta-7.2.2-0) over TLS secured channel with ESMTP id 1b512825.0.20346.00-368.29943.p02c11o148.mxlogic.net (envelope-from ); Tue, 12 Nov 2013 04:49:13 -0700 (MST) X-MXL-Hash: 528215b91206d2d8-f6274e7c4fa8482a2836ebaecdb14c5e2d1ede6b Received: from ex-mb1.corp.adtran.com ([fe80::51a3:972d:5f16:9952]) by ex-hc1.corp.adtran.com ([fe80::a43f:7ea6:7688:37b%13]) with mapi id 14.01.0438.000; Tue, 12 Nov 2013 05:49:05 -0600 From: KEN KO To: "Romascanu, Dan (Dan)" , "Bugenhagen, Michael K" , "lmap@ietf.org" , 'Trevor Burbridge' , "'philip.eardley@bt.com'" , 'marcelo bagnulo braun' Thread-Topic: Comments & opinions on Draft - RE: [lmap] call for consensus - adoption of draft-burbridge-lmap-information-model as WG document Thread-Index: Ac7bVGevh+4pI7FlSz+236qyIybWMQDqGOVAACVfjoAAArUz8A== Date: Tue, 12 Nov 2013 11:49:05 +0000 Message-ID: References: <9904FB1B0159DA42B0B887B7FA8119CA1292EAA3@AZ-FFEXMB04.global.avaya.com> <9904FB1B0159DA42B0B887B7FA8119CA1293A32C@AZ-FFEXMB04.global.avaya.com> In-Reply-To: <9904FB1B0159DA42B0B887B7FA8119CA1293A32C@AZ-FFEXMB04.global.avaya.com> Accept-Language: en-US Content-Language: en-US X-MS-Has-Attach: X-MS-TNEF-Correlator: x-originating-ip: [172.30.5.205] Content-Type: text/plain; charset="us-ascii" Content-Transfer-Encoding: quoted-printable MIME-Version: 1.0 X-AnalysisOut: [v=2.0 cv=BYNvJMR2 c=1 sm=1 a=0XgpNN2/4a34ymu16SVwsQ==:17 a] X-AnalysisOut: [=CB9cjBeVwSIA:10 a=l7oVkYRohz0A:10 a=qZHQZMT3apYA:10 a=BLc] X-AnalysisOut: [eEmwcHowA:10 a=kj9zAlcOel0A:10 a=xqWC_Br6kY4A:10 a=eJNrpio] X-AnalysisOut: [GAAAA:8 a=u2XNE98oYDgA:10 a=gEsRDbIwAAAA:8 a=48vgC7mUAAAA:] X-AnalysisOut: [8 a=e9qsufxtAAAA:8 a=J_N6KFswAAAA:8 a=6HurVjqktKOu7hD588oA] X-AnalysisOut: [:9 a=CjuIK1q_8ugA:10 a=4dyfk4FV-b8A:10 a=lZB815dzVvQA:10 a] X-AnalysisOut: [=W1qU_X6G3J8A:10 a=Pwbduc0PQ3sA:10 a=SHRPBMo7DZWbEYuZ:21 a] X-AnalysisOut: [=J5F5kSwUktUjpnKv:21] X-Spam: [F=0.2000000000; CM=0.500; S=0.200(2010122901)] X-MAIL-FROM: X-SOURCE-IP: [76.164.174.81] Cc: "'Stark, Barbara'" , "Cook, Charles" , "'MORTON, ALFRED C \(AL\)'" Subject: Re: [lmap] Comments & opinions on Draft - RE: call for consensus - adoption of draft-burbridge-lmap-information-model as WG document X-BeenThere: lmap@ietf.org X-Mailman-Version: 2.1.12 Precedence: list List-Id: Large Scale Measurement of Access network Performance List-Unsubscribe: , List-Archive: List-Post: List-Help: List-Subscribe: , X-List-Received-Date: Tue, 12 Nov 2013 11:49:33 -0000 Getting back to the original question as Dan asked it, I support adopting d= raft-burbridge-lmap-information-model as a WG document. Thanks, Ken -----Original Message----- From: Romascanu, Dan (Dan) [mailto:dromasca@avaya.com]=20 Sent: Tuesday, November 12, 2013 5:39 AM To: Bugenhagen, Michael K; lmap@ietf.org; 'Trevor Burbridge'; 'philip.eardl= ey@bt.com'; 'marcelo bagnulo braun' Cc: 'MORTON, ALFRED C (AL)'; Cook, Charles; KEN KO; 'Stark, Barbara' Subject: RE: Comments & opinions on Draft - RE: [lmap] call for consensus -= adoption of draft-burbridge-lmap-information-model as WG document Hi Michael, Thank you for your comments. As I can see the authors started to address th= em.=20 My question (wearing the WG chair hat :-) ) - are you OK with the adoption = of draft-burbridge-lmap-information-model as WG document? Saying 'no' means= that you consider that the document needs quite serious changes before bec= oming a WG document, maybe that you have or you are aware about one ore mor= e alternate proposals to be submitted. Saying 'yes' means the document is o= n the right direction, with changes and improvements that can be made in th= e following iterations.=20 Thanks and Regards, Dan > -----Original Message----- > From: Bugenhagen, Michael K > [mailto:Michael.K.Bugenhagen@centurylink.com] > Sent: Monday, November 11, 2013 7:05 PM > To: Romascanu, Dan (Dan); lmap@ietf.org; 'Trevor Burbridge';=20 > 'philip.eardley@bt.com'; 'marcelo bagnulo braun' > Cc: 'MORTON, ALFRED C (AL)'; Cook, Charles; 'KEN KO'; 'Stark, Barbara' > Subject: Comments & opinions on Draft - RE: [lmap] call for consensus=20 > - adoption of draft-burbridge-lmap-information-model as WG document >=20 > Dan, >=20 > Comment 1. Is written from an ISP Operational team perspective (Scope=20 > says ISP use case so... ) Comment 2. Is future proofing the Registry=20 > concept but it also is in the draft a bit.. >=20 >=20 > 1) ISP use case requirements - (The yet another test system overlay > issue) >=20 > ISP's have multiple IP based "testing systems" / "Measurement=20 > Agents" deployed today, many of which don't co-exist well. > So one ongoing fairly solid ISP requirement that the groups should=20 > strongly consider is the ability of a MA to be open flexible framework=20 > MA that can run both Op's ADHOC test, and performance testing without=20 > two MA's. Example (a MA / probe that can run VoIP, VIDEO, LMAP, DNS=20 > testing, ..... vs. multiple probes doing one each.) >=20 > ** Current draft that means adding the ability to >=20 > ADHOC / Multi-user op's needs for considerations > a) MA capability to both Push and Pull test results > b) MA Radius / Authentication to allow item a) > c) Multiple test capability (OAM, and throughput at the same time by=20 > different Op's teams) > d) Resource availability checking (when you can run more than one test=20 > at a time) > e) MA to MA test capability negotiation (if you are working with a=20 > customers or 3rd party probe) >=20 >=20 >=20 > 2) Future proof the Registry / MA probe concept into Virtual=20 > Applications - (start thinking Virtualization Service Catalog) >=20 > As NFV (network functional virtualization) will likely become=20 > available by the time some of this is being implemented we should=20 > consider adding a "hypervisor" type to the Registry. The thought here=20 > is that Virtual applications come in a default deployable set, *.OVF > files for VMware, ... for KVM, and so on. ** ONE KEY to service > catalogs is a trusted source for the code, one should NEVER (NEVER - > EVER) deploy a virtual appliance from a public source on their=20 > network... >=20 > I think the registry could be a good solution to this issue - provided=20 > we place a high level or safety checks on anything getting loaded into=20 > the registry itself. >=20 >=20 >=20 >=20 >=20 >=20 >=20 >=20 >=20 >=20 >=20 >=20 >=20 >=20 >=20 >=20 >=20 >=20 >=20 >=20 >=20 >=20 >=20 >=20 >=20 >=20 >=20 >=20 >=20 >=20 >=20 >=20 >=20 > -----Original Message----- > From: lmap-bounces@ietf.org [mailto:lmap-bounces@ietf.org] On Behalf=20 > Of Romascanu, Dan (Dan) > Sent: Wednesday, November 06, 2013 6:58 PM > To: lmap@ietf.org > Subject: [lmap] call for consensus - adoption of draft-burbridge-lmap-=20 > information-model as WG document >=20 >=20 > Today in the LMAP meeting we ran a hum on the question whether we=20 > should adopt draft-burbridge-lmap-information-model. The result was=20 > read as showing consensus in favor of the adoption, although there was=20 > at least one opposed opinion expressed by Benoit (as contributor). >=20 > We would like to extend this call to the participants who could not=20 > attend the meeting. Please send your opinions on this question to the=20 > WG mail list before Friday 11/15. >=20 > Thanks and Regards, >=20 > Dan >=20 >=20 >=20 > _______________________________________________ > lmap mailing list > lmap@ietf.org > https://www.ietf.org/mailman/listinfo/lmap From Michael.K.Bugenhagen@centurylink.com Tue Nov 12 07:48:47 2013 Return-Path: X-Original-To: lmap@ietfa.amsl.com Delivered-To: lmap@ietfa.amsl.com Received: from localhost (localhost [127.0.0.1]) by ietfa.amsl.com (Postfix) with ESMTP id 9DF0221E825A for ; Tue, 12 Nov 2013 07:48:47 -0800 (PST) X-Virus-Scanned: amavisd-new at amsl.com X-Spam-Flag: NO X-Spam-Score: -2.599 X-Spam-Level: X-Spam-Status: No, score=-2.599 tagged_above=-999 required=5 tests=[AWL=0.000, BAYES_00=-2.599] Received: from mail.ietf.org ([12.22.58.30]) by localhost (ietfa.amsl.com [127.0.0.1]) (amavisd-new, port 10024) with ESMTP id T+mHA0qegIN1 for ; Tue, 12 Nov 2013 07:48:42 -0800 (PST) Received: from suomp64i.qwest.com (suomp64i.qwest.com [155.70.16.237]) by ietfa.amsl.com (Postfix) with ESMTP id 98ED521E8249 for ; Tue, 12 Nov 2013 07:48:27 -0800 (PST) Received: from lxdenvmpc030.qintra.com (lxdenvmpc030.qintra.com [10.1.51.30]) by suomp64i.qwest.com (8.14.4/8.14.4) with ESMTP id rACFmCpS023079 (version=TLSv1/SSLv3 cipher=DHE-RSA-AES256-SHA bits=256 verify=NO); Tue, 12 Nov 2013 09:48:12 -0600 (CST) Received: from lxdenvmpc030.qintra.com (unknown [127.0.0.1]) by IMSA (Postfix) with ESMTP id E05231E008A; Tue, 12 Nov 2013 08:48:06 -0700 (MST) Received: from suomp61i.qintra.com (unknown [151.119.91.93]) by lxdenvmpc030.qintra.com (Postfix) with ESMTP id A691D1E0079; Tue, 12 Nov 2013 08:48:06 -0700 (MST) Received: from suomp61i.qintra.com (localhost [127.0.0.1]) by suomp61i.qintra.com (8.14.4/8.14.4) with ESMTP id rACFm5u5006228; Tue, 12 Nov 2013 09:48:05 -0600 (CST) Received: from vodcwhubex501.ctl.intranet (vodcwhubex501.ctl.intranet [151.117.206.27]) by suomp61i.qintra.com (8.14.4/8.14.4) with ESMTP id rACFm05g005808 (version=TLSv1/SSLv3 cipher=AES128-SHA bits=128 verify=FAIL); Tue, 12 Nov 2013 09:48:05 -0600 (CST) Received: from PODCWMBXEX505.ctl.intranet ([fe80::f87e:fe44:ad72:b610]) by vodcwhubex501.ctl.intranet ([151.117.206.27]) with mapi id 14.02.0318.001; Tue, 12 Nov 2013 09:48:00 -0600 From: "Bugenhagen, Michael K" To: "'Romascanu, Dan (Dan)'" , "lmap@ietf.org" , "'Trevor Burbridge'" , "'philip.eardley@bt.com'" , "'marcelo bagnulo braun'" Thread-Topic: Comments & opinions on Draft - RE: [lmap] call for consensus - adoption of draft-burbridge-lmap-information-model as WG document Thread-Index: Ac7bVGevh+4pI7FlSz+236qyIybWMQDqGOVAACVfjoAACwp8EA== Date: Tue, 12 Nov 2013 15:47:59 +0000 Message-ID: References: <9904FB1B0159DA42B0B887B7FA8119CA1292EAA3@AZ-FFEXMB04.global.avaya.com> <9904FB1B0159DA42B0B887B7FA8119CA1293A32C@AZ-FFEXMB04.global.avaya.com> In-Reply-To: <9904FB1B0159DA42B0B887B7FA8119CA1293A32C@AZ-FFEXMB04.global.avaya.com> Accept-Language: en-US Content-Language: en-US X-MS-Has-Attach: X-MS-TNEF-Correlator: x-originating-ip: [151.117.206.7] Content-Type: text/plain; charset="us-ascii" Content-Transfer-Encoding: quoted-printable MIME-Version: 1.0 X-CFilter-Loop: Reflected Cc: "'Stark, Barbara'" , "Cook, Charles" , "'MORTON, ALFRED C \(AL\)'" , 'KEN KO' Subject: Re: [lmap] Comments & opinions on Draft - RE: call for consensus - adoption of draft-burbridge-lmap-information-model as WG document X-BeenThere: lmap@ietf.org X-Mailman-Version: 2.1.12 Precedence: list List-Id: Large Scale Measurement of Access network Performance List-Unsubscribe: , List-Archive: List-Post: List-Help: List-Subscribe: , X-List-Received-Date: Tue, 12 Nov 2013 15:48:47 -0000 Dan -=20 Thanks - you are of course correct, let me re-read and answer that one di= rectly. I'll get that out today.... Mike -----Original Message----- From: Romascanu, Dan (Dan) [mailto:dromasca@avaya.com]=20 Sent: Tuesday, November 12, 2013 4:39 AM To: Bugenhagen, Michael K; lmap@ietf.org; 'Trevor Burbridge'; 'philip.eardl= ey@bt.com'; 'marcelo bagnulo braun' Cc: 'MORTON, ALFRED C (AL)'; Cook, Charles; 'KEN KO'; 'Stark, Barbara' Subject: RE: Comments & opinions on Draft - RE: [lmap] call for consensus -= adoption of draft-burbridge-lmap-information-model as WG document Hi Michael, Thank you for your comments. As I can see the authors started to address th= em.=20 My question (wearing the WG chair hat :-) ) - are you OK with the adoption = of draft-burbridge-lmap-information-model as WG document? Saying 'no' means= that you consider that the document needs quite serious changes before bec= oming a WG document, maybe that you have or you are aware about one ore mor= e alternate proposals to be submitted. Saying 'yes' means the document is o= n the right direction, with changes and improvements that can be made in th= e following iterations.=20 Thanks and Regards, Dan > -----Original Message----- > From: Bugenhagen, Michael K > [mailto:Michael.K.Bugenhagen@centurylink.com] > Sent: Monday, November 11, 2013 7:05 PM > To: Romascanu, Dan (Dan); lmap@ietf.org; 'Trevor Burbridge';=20 > 'philip.eardley@bt.com'; 'marcelo bagnulo braun' > Cc: 'MORTON, ALFRED C (AL)'; Cook, Charles; 'KEN KO'; 'Stark, Barbara' > Subject: Comments & opinions on Draft - RE: [lmap] call for consensus=20 > - adoption of draft-burbridge-lmap-information-model as WG document >=20 > Dan, >=20 > Comment 1. Is written from an ISP Operational team perspective (Scope=20 > says ISP use case so... ) Comment 2. Is future proofing the Registry=20 > concept but it also is in the draft a bit.. >=20 >=20 > 1) ISP use case requirements - (The yet another test system overlay > issue) >=20 > ISP's have multiple IP based "testing systems" / "Measurement=20 > Agents" deployed today, many of which don't co-exist well. > So one ongoing fairly solid ISP requirement that the groups should=20 > strongly consider is the ability of a MA to be open flexible framework=20 > MA that can run both Op's ADHOC test, and performance testing without=20 > two MA's. Example (a MA / probe that can run VoIP, VIDEO, LMAP, DNS=20 > testing, ..... vs. multiple probes doing one each.) >=20 > ** Current draft that means adding the ability to >=20 > ADHOC / Multi-user op's needs for considerations > a) MA capability to both Push and Pull test results > b) MA Radius / Authentication to allow item a) > c) Multiple test capability (OAM, and throughput at the same time by=20 > different Op's teams) > d) Resource availability checking (when you can run more than one test=20 > at a time) > e) MA to MA test capability negotiation (if you are working with a=20 > customers or 3rd party probe) >=20 >=20 >=20 > 2) Future proof the Registry / MA probe concept into Virtual=20 > Applications - (start thinking Virtualization Service Catalog) >=20 > As NFV (network functional virtualization) will likely become=20 > available by the time some of this is being implemented we should=20 > consider adding a "hypervisor" type to the Registry. The thought here=20 > is that Virtual applications come in a default deployable set, *.OVF > files for VMware, ... for KVM, and so on. ** ONE KEY to service > catalogs is a trusted source for the code, one should NEVER (NEVER - > EVER) deploy a virtual appliance from a public source on their=20 > network... >=20 > I think the registry could be a good solution to this issue - provided=20 > we place a high level or safety checks on anything getting loaded into=20 > the registry itself. >=20 >=20 >=20 >=20 >=20 >=20 >=20 >=20 >=20 >=20 >=20 >=20 >=20 >=20 >=20 >=20 >=20 >=20 >=20 >=20 >=20 >=20 >=20 >=20 >=20 >=20 >=20 >=20 >=20 >=20 >=20 >=20 >=20 > -----Original Message----- > From: lmap-bounces@ietf.org [mailto:lmap-bounces@ietf.org] On Behalf=20 > Of Romascanu, Dan (Dan) > Sent: Wednesday, November 06, 2013 6:58 PM > To: lmap@ietf.org > Subject: [lmap] call for consensus - adoption of draft-burbridge-lmap-=20 > information-model as WG document >=20 >=20 > Today in the LMAP meeting we ran a hum on the question whether we=20 > should adopt draft-burbridge-lmap-information-model. The result was=20 > read as showing consensus in favor of the adoption, although there was=20 > at least one opposed opinion expressed by Benoit (as contributor). >=20 > We would like to extend this call to the participants who could not=20 > attend the meeting. Please send your opinions on this question to the=20 > WG mail list before Friday 11/15. >=20 > Thanks and Regards, >=20 > Dan >=20 >=20 >=20 > _______________________________________________ > lmap mailing list > lmap@ietf.org > https://www.ietf.org/mailman/listinfo/lmap From bs7652@att.com Tue Nov 12 08:18:01 2013 Return-Path: X-Original-To: lmap@ietfa.amsl.com Delivered-To: lmap@ietfa.amsl.com Received: from localhost (localhost [127.0.0.1]) by ietfa.amsl.com (Postfix) with ESMTP id 7FF8511E818E for ; Tue, 12 Nov 2013 08:18:01 -0800 (PST) X-Virus-Scanned: amavisd-new at amsl.com X-Spam-Flag: NO X-Spam-Score: -6.599 X-Spam-Level: X-Spam-Status: No, score=-6.599 tagged_above=-999 required=5 tests=[BAYES_00=-2.599, RCVD_IN_DNSWL_MED=-4] Received: from mail.ietf.org ([12.22.58.30]) by localhost (ietfa.amsl.com [127.0.0.1]) (amavisd-new, port 10024) with ESMTP id pVwyskiuNhmB for ; Tue, 12 Nov 2013 08:17:38 -0800 (PST) Received: from nbfkord-smmo05.seg.att.com (nbfkord-smmo05.seg.att.com [209.65.160.92]) by ietfa.amsl.com (Postfix) with ESMTP id 6A22421F8D62 for ; Tue, 12 Nov 2013 08:17:25 -0800 (PST) Received: from unknown [144.160.229.23] (EHLO alpi154.enaf.aldc.att.com) by nbfkord-smmo05.seg.att.com(mxl_mta-6.15.0-1) over TLS secured channel with ESMTP id 49452825.0.6592560.00-372.18428933.nbfkord-smmo05.seg.att.com (envelope-from ); Tue, 12 Nov 2013 16:17:25 +0000 (UTC) X-MXL-Hash: 52825495529cf86f-ee6921b07b2d71cdddc8855561a2514a2b6c4e1b Received: from enaf.aldc.att.com (localhost [127.0.0.1]) by alpi154.enaf.aldc.att.com (8.14.5/8.14.5) with ESMTP id rACGHO2Q018119 for ; Tue, 12 Nov 2013 11:17:24 -0500 Received: from alpi133.aldc.att.com (alpi133.aldc.att.com [130.8.217.3]) by alpi154.enaf.aldc.att.com (8.14.5/8.14.5) with ESMTP id rACGHIxQ018041 (version=TLSv1/SSLv3 cipher=DHE-RSA-AES256-SHA bits=256 verify=NO) for ; Tue, 12 Nov 2013 11:17:20 -0500 Received: from GAALPA1MSGHUB9C.ITServices.sbc.com (GAALPA1MSGHUB9C.itservices.sbc.com [130.8.36.89]) by alpi133.aldc.att.com (RSA Interceptor) for ; Tue, 12 Nov 2013 16:17:04 GMT Received: from GAALPA1MSGUSR9L.ITServices.sbc.com ([130.8.36.69]) by GAALPA1MSGHUB9C.ITServices.sbc.com ([130.8.36.89]) with mapi id 14.03.0158.001; Tue, 12 Nov 2013 11:17:04 -0500 From: "STARK, BARBARA H" To: "lmap@ietf.org" Thread-Topic: Measurement Suppression broadcasting and non-acknowledgement Thread-Index: Ac7fwpzx+PvU5I6kRIq1pPQs5SSV/w== Date: Tue, 12 Nov 2013 16:17:01 +0000 Message-ID: <2D09D61DDFA73D4C884805CC7865E61130399373@GAALPA1MSGUSR9L.ITServices.sbc.com> Accept-Language: en-US Content-Language: en-US X-MS-Has-Attach: X-MS-TNEF-Correlator: x-originating-ip: [135.70.12.184] Content-Type: text/plain; charset="us-ascii" Content-Transfer-Encoding: quoted-printable MIME-Version: 1.0 X-RSA-Inspected: yes X-RSA-Classifications: public X-Spam: [F=0.2000000000; CM=0.500; S=0.200(2010122901)] X-MAIL-FROM: X-SOURCE-IP: [144.160.229.23] X-AnalysisOut: [v=2.0 cv=HdWjuF48 c=1 sm=0 a=VXHOiMMwGAwA+y4G3/O+aw==:17 a] X-AnalysisOut: [=ddS1sNSY6owA:10 a=4kUb2ayGZuAA:10 a=ofMgfj31e3cA:10 a=BLc] X-AnalysisOut: [eEmwcHowA:10 a=kj9zAlcOel0A:10 a=zQP7CpKOAAAA:8 a=XIqpo32R] X-AnalysisOut: [AAAA:8 a=bvZXIkwaAWEA:10 a=AzRhLLcE4IEi7RPv6lcA:9 a=CjuIK1] X-AnalysisOut: [q_8ugA:10] Subject: [lmap] Measurement Suppression broadcasting and non-acknowledgement X-BeenThere: lmap@ietf.org X-Mailman-Version: 2.1.12 Precedence: list List-Id: Large Scale Measurement of Access network Performance List-Unsubscribe: , List-Archive: List-Post: List-Help: List-Subscribe: , X-List-Received-Date: Tue, 12 Nov 2013 16:18:01 -0000 I have some concerns with the way "Measurement Suppression" is architected = in framework currently. framework says "the "suppress" message is not acknowledged, since it is lik= ely to be broadcast to several /many MAs at a time when the measurement sys= tem wants to eliminate inessential traffic." 1. I'm not sure if this is intended to literally mean "broadcast" (which is= a very specific concept in IP that uses broadcast destination IP addresses= like 255.255.255.255 or more specific subnet addresses to reach all hosts = on a network with a single sent message), or "multicast" (similar to broadc= ast but sent to a multicast address), or just massive quantities of unicast= messages sent "simultaneously". Where MAs are internal to an enterprise or campus network (managed by Contr= ollers also internal to the network), I could see where literal IP broadcas= t messages (or even multicast messages) might be a good technique to use. F= or MAs managed by Controllers that are out on the Internet somewhere (which= I'd hope we all agree are definitely in scope), I don't think it's practic= al. Therefore I disagree that it's likely. 2. I believe it would be a bad idea for MAs managed by Controllers out on t= he Internet to react to a suppression message that is unauthenticated and u= nencrypted. Therefore, there exists a need for it to be possible for the Co= ntroller and each MA to establish authenticated and encrypted (e.g., TLS) s= essions prior to the Controller sending the suppression command. 3. The overhead of having the MA send an Ack message after setting up the a= uthenticated and encrypted session isn't so big that it should be prohibite= d, if the Controller operator wants such an Ack to occur. Therefore, I disagree that the framework should mandate non-acknowledgement= of suppression messages. I have no problem with unacknowledged suppression= messages being allowed -- this can be useful in enterprise/campus environm= ents. But I also believe there is value in supporting acknowledged suppres= sion messages, and that this option should also be allowed. From j.schoenwaelder@jacobs-university.de Tue Nov 12 08:35:16 2013 Return-Path: X-Original-To: lmap@ietfa.amsl.com Delivered-To: lmap@ietfa.amsl.com Received: from localhost (localhost [127.0.0.1]) by ietfa.amsl.com (Postfix) with ESMTP id 4DCF221E8257 for ; Tue, 12 Nov 2013 08:35:16 -0800 (PST) X-Virus-Scanned: amavisd-new at amsl.com X-Spam-Flag: NO X-Spam-Score: -103.203 X-Spam-Level: X-Spam-Status: No, score=-103.203 tagged_above=-999 required=5 tests=[AWL=0.046, BAYES_00=-2.599, HELO_EQ_DE=0.35, RCVD_IN_DNSWL_LOW=-1, USER_IN_WHITELIST=-100] Received: from mail.ietf.org ([12.22.58.30]) by localhost (ietfa.amsl.com [127.0.0.1]) (amavisd-new, port 10024) with ESMTP id LPmWejaWSFHe for ; Tue, 12 Nov 2013 08:35:11 -0800 (PST) Received: from hermes.jacobs-university.de (hermes.jacobs-university.de [212.201.44.23]) by ietfa.amsl.com (Postfix) with ESMTP id 2839421E8210 for ; Tue, 12 Nov 2013 08:35:08 -0800 (PST) Received: from localhost (demetrius2.jacobs-university.de [212.201.44.47]) by hermes.jacobs-university.de (Postfix) with ESMTP id 8FF0720047; Tue, 12 Nov 2013 17:35:07 +0100 (CET) X-Virus-Scanned: amavisd-new at jacobs-university.de Received: from hermes.jacobs-university.de ([212.201.44.23]) by localhost (demetrius2.jacobs-university.de [212.201.44.32]) (amavisd-new, port 10024) with ESMTP id F03uqK0em75v; Tue, 12 Nov 2013 17:35:07 +0100 (CET) Received: from elstar.local (elstar.jacobs.jacobs-university.de [10.50.231.133]) by hermes.jacobs-university.de (Postfix) with ESMTP id 1BDEA20076; Tue, 12 Nov 2013 17:35:04 +0100 (CET) Received: by elstar.local (Postfix, from userid 501) id 6416C2946106; Tue, 12 Nov 2013 17:35:00 +0100 (CET) Date: Tue, 12 Nov 2013 17:35:00 +0100 From: Juergen Schoenwaelder To: "STARK, BARBARA H" Message-ID: <20131112163500.GD3992@elstar.local> Mail-Followup-To: "STARK, BARBARA H" , "lmap@ietf.org" References: <2D09D61DDFA73D4C884805CC7865E61130399373@GAALPA1MSGUSR9L.ITServices.sbc.com> MIME-Version: 1.0 Content-Type: text/plain; charset=us-ascii Content-Disposition: inline In-Reply-To: <2D09D61DDFA73D4C884805CC7865E61130399373@GAALPA1MSGUSR9L.ITServices.sbc.com> User-Agent: Mutt/1.5.21 (2010-09-15) Cc: "lmap@ietf.org" Subject: Re: [lmap] Measurement Suppression broadcasting and non-acknowledgement X-BeenThere: lmap@ietf.org X-Mailman-Version: 2.1.12 Precedence: list Reply-To: Juergen Schoenwaelder List-Id: Large Scale Measurement of Access network Performance List-Unsubscribe: , List-Archive: List-Post: List-Help: List-Subscribe: , X-List-Received-Date: Tue, 12 Nov 2013 16:35:16 -0000 Barbara, I share your concerns. /js On Tue, Nov 12, 2013 at 04:17:01PM +0000, STARK, BARBARA H wrote: > I have some concerns with the way "Measurement Suppression" is architected in framework currently. > > framework says "the "suppress" message is not acknowledged, since it is likely to be broadcast to several /many MAs at a time when the measurement system wants to eliminate inessential traffic." > > 1. I'm not sure if this is intended to literally mean "broadcast" (which is a very specific concept in IP that uses broadcast destination IP addresses like 255.255.255.255 or more specific subnet addresses to reach all hosts on a network with a single sent message), or "multicast" (similar to broadcast but sent to a multicast address), or just massive quantities of unicast messages sent "simultaneously". > Where MAs are internal to an enterprise or campus network (managed by Controllers also internal to the network), I could see where literal IP broadcast messages (or even multicast messages) might be a good technique to use. For MAs managed by Controllers that are out on the Internet somewhere (which I'd hope we all agree are definitely in scope), I don't think it's practical. Therefore I disagree that it's likely. > > 2. I believe it would be a bad idea for MAs managed by Controllers out on the Internet to react to a suppression message that is unauthenticated and unencrypted. Therefore, there exists a need for it to be possible for the Controller and each MA to establish authenticated and encrypted (e.g., TLS) sessions prior to the Controller sending the suppression command. > > 3. The overhead of having the MA send an Ack message after setting up the authenticated and encrypted session isn't so big that it should be prohibited, if the Controller operator wants such an Ack to occur. > > Therefore, I disagree that the framework should mandate non-acknowledgement of suppression messages. I have no problem with unacknowledged suppression messages being allowed -- this can be useful in enterprise/campus environments. But I also believe there is value in supporting acknowledged suppression messages, and that this option should also be allowed. > _______________________________________________ > lmap mailing list > lmap@ietf.org > https://www.ietf.org/mailman/listinfo/lmap -- Juergen Schoenwaelder Jacobs University Bremen gGmbH Phone: +49 421 200 3587 Campus Ring 1, 28759 Bremen, Germany Fax: +49 421 200 3103 From bs7652@att.com Tue Nov 12 08:50:40 2013 Return-Path: X-Original-To: lmap@ietfa.amsl.com Delivered-To: lmap@ietfa.amsl.com Received: from localhost (localhost [127.0.0.1]) by ietfa.amsl.com (Postfix) with ESMTP id 11B0921E81BF for ; Tue, 12 Nov 2013 08:50:40 -0800 (PST) X-Virus-Scanned: amavisd-new at amsl.com X-Spam-Flag: NO X-Spam-Score: -6.599 X-Spam-Level: X-Spam-Status: No, score=-6.599 tagged_above=-999 required=5 tests=[BAYES_00=-2.599, RCVD_IN_DNSWL_MED=-4] Received: from mail.ietf.org ([12.22.58.30]) by localhost (ietfa.amsl.com [127.0.0.1]) (amavisd-new, port 10024) with ESMTP id up5a81WJ929N for ; Tue, 12 Nov 2013 08:50:30 -0800 (PST) Received: from nbfkord-smmo07.seg.att.com (nbfkord-smmo07.seg.att.com [209.65.160.93]) by ietfa.amsl.com (Postfix) with ESMTP id 933A421E81D1 for ; Tue, 12 Nov 2013 08:50:21 -0800 (PST) Received: from unknown [144.160.229.23] (EHLO alpi154.enaf.aldc.att.com) by nbfkord-smmo07.seg.att.com(mxl_mta-6.15.0-1) over TLS secured channel with ESMTP id d4c52825.0.346340.00-396.970935.nbfkord-smmo07.seg.att.com (envelope-from ); Tue, 12 Nov 2013 16:50:21 +0000 (UTC) X-MXL-Hash: 52825c4d26549c0a-ac19b216c19770cd1e881cb0bf33c2f2f1903e24 Received: from enaf.aldc.att.com (localhost [127.0.0.1]) by alpi154.enaf.aldc.att.com (8.14.5/8.14.5) with ESMTP id rACGoKtw025169 for ; Tue, 12 Nov 2013 11:50:20 -0500 Received: from alpi133.aldc.att.com (alpi133.aldc.att.com [130.8.217.3]) by alpi154.enaf.aldc.att.com (8.14.5/8.14.5) with ESMTP id rACGoDAQ025095 (version=TLSv1/SSLv3 cipher=DHE-RSA-AES256-SHA bits=256 verify=NO) for ; Tue, 12 Nov 2013 11:50:15 -0500 Received: from GAALPA1MSGHUB9E.ITServices.sbc.com (GAALPA1MSGHUB9E.itservices.sbc.com [130.8.36.91]) by alpi133.aldc.att.com (RSA Interceptor) for ; Tue, 12 Nov 2013 16:50:07 GMT Received: from GAALPA1MSGUSR9L.ITServices.sbc.com ([130.8.36.69]) by GAALPA1MSGHUB9E.ITServices.sbc.com ([130.8.36.91]) with mapi id 14.03.0158.001; Tue, 12 Nov 2013 11:50:07 -0500 From: "STARK, BARBARA H" To: "lmap@ietf.org" Thread-Topic: degrees of Measurement Suppression Thread-Index: Ac7fxy6a9B101VqJQi6oteMX5LjSDg== Date: Tue, 12 Nov 2013 16:50:06 +0000 Message-ID: <2D09D61DDFA73D4C884805CC7865E611303993CD@GAALPA1MSGUSR9L.ITServices.sbc.com> Accept-Language: en-US Content-Language: en-US X-MS-Has-Attach: X-MS-TNEF-Correlator: x-originating-ip: [135.70.12.184] Content-Type: text/plain; charset="us-ascii" Content-Transfer-Encoding: quoted-printable MIME-Version: 1.0 X-RSA-Inspected: yes X-RSA-Classifications: public X-Spam: [F=0.2000000000; CM=0.500; S=0.200(2010122901)] X-MAIL-FROM: X-SOURCE-IP: [144.160.229.23] X-AnalysisOut: [v=2.0 cv=HsnB7TvS c=1 sm=0 a=VXHOiMMwGAwA+y4G3/O+aw==:17 a] X-AnalysisOut: [=ddS1sNSY6owA:10 a=ofMgfj31e3cA:10 a=BLceEmwcHowA:10 a=kj9] X-AnalysisOut: [zAlcOel0A:10 a=zQP7CpKOAAAA:8 a=XIqpo32RAAAA:8 a=qESuIFSon] X-AnalysisOut: [Q8A:10 a=SlY4ubuk2L_lfjsmqhUA:9 a=CjuIK1q_8ugA:10] Subject: [lmap] degrees of Measurement Suppression X-BeenThere: lmap@ietf.org X-Mailman-Version: 2.1.12 Precedence: list List-Id: Large Scale Measurement of Access network Performance List-Unsubscribe: , List-Archive: List-Post: List-Help: List-Subscribe: , X-List-Received-Date: Tue, 12 Nov 2013 16:50:40 -0000 A number of providers have discussed a model of Measurement Suppression tha= t supports more granular degrees of suppression (other than just suppress a= nd don't suppress). Michael Bugenhagen presented the original version of th= is model, and he said it would be ok if I brought it up to IETF. Following are highlights of the proposal (with some modifications I've incl= uded as a result of additional discussion): 1. Measurement Methods (or perhaps Tasks) have a configurable parameter tha= t indicates whether the Controller operator considers them to be "critical = for OAM", "not critical, but not resource intensive", and "not critical and= resource intensive". 2. The Controller can specify degrees of Measurement Suppression, which sho= uld include: halt all non-critical tasks immediately but allow OAM tasks; h= alt resource-intensive tasks immediately but allow non-resource-intensive t= asks; finish resource intensive tasks but do not start new resource-intensi= ve tasks and allow all non-resource-intensive tasks; allow all tasks. 3. It may also be possible for the unspecified bootstrap mechanism to instr= uct the MA to suppress measurements. Where multiple channels exist for Meas= urement Suppression (e.g., Controller and bootstrap), the MA is to use the = most restrictive setting. Barbara From Michael.K.Bugenhagen@centurylink.com Tue Nov 12 07:45:03 2013 Return-Path: X-Original-To: lmap@ietfa.amsl.com Delivered-To: lmap@ietfa.amsl.com Received: from localhost (localhost [127.0.0.1]) by ietfa.amsl.com (Postfix) with ESMTP id 4941C21E8242 for ; Tue, 12 Nov 2013 07:45:03 -0800 (PST) X-Virus-Scanned: amavisd-new at amsl.com X-Spam-Flag: NO X-Spam-Score: -2.599 X-Spam-Level: X-Spam-Status: No, score=-2.599 tagged_above=-999 required=5 tests=[AWL=0.000, BAYES_00=-2.599] Received: from mail.ietf.org ([12.22.58.30]) by localhost (ietfa.amsl.com [127.0.0.1]) (amavisd-new, port 10024) with ESMTP id 8DakOTmx2TqH for ; Tue, 12 Nov 2013 07:44:57 -0800 (PST) Received: from sudnp799.qwest.com (sudnp799.qwest.com [155.70.32.99]) by ietfa.amsl.com (Postfix) with ESMTP id 78F9411E820B for ; Tue, 12 Nov 2013 07:43:43 -0800 (PST) Received: from lxomavmpc030.qintra.com (emailout.qintra.com [151.117.207.30]) by sudnp799.qwest.com (8.14.4/8.14.4) with ESMTP id rACFhW1k019252 (version=TLSv1/SSLv3 cipher=DHE-RSA-AES256-SHA bits=256 verify=NO); Tue, 12 Nov 2013 08:43:33 -0700 (MST) Received: from lxomavmpc030.qintra.com (unknown [127.0.0.1]) by IMSA (Postfix) with ESMTP id 7CE601E004F; Tue, 12 Nov 2013 09:43:27 -0600 (CST) Received: from suomp60i.qintra.com (unknown [10.6.10.61]) by lxomavmpc030.qintra.com (Postfix) with ESMTP id 55B4F1E0078; Tue, 12 Nov 2013 09:43:27 -0600 (CST) Received: from suomp60i.qintra.com (localhost [127.0.0.1]) by suomp60i.qintra.com (8.14.4/8.14.4) with ESMTP id rACFhQ1A003499; Tue, 12 Nov 2013 09:43:26 -0600 (CST) Received: from vodcwhubex502.ctl.intranet (vodcwhubex502.ctl.intranet [151.117.206.28]) by suomp60i.qintra.com (8.14.4/8.14.4) with ESMTP id rACFhQNK003487 (version=TLSv1/SSLv3 cipher=AES128-SHA bits=128 verify=FAIL); Tue, 12 Nov 2013 09:43:26 -0600 (CST) Received: from PODCWMBXEX505.ctl.intranet ([fe80::f87e:fe44:ad72:b610]) by vodcwhubex502.ctl.intranet ([2002:9775:ce1c::9775:ce1c]) with mapi id 14.02.0318.001; Tue, 12 Nov 2013 09:43:26 -0600 From: "Bugenhagen, Michael K" To: "'Juergen Schoenwaelder'" Thread-Topic: [lmap] Comments & opinions on Draft - RE: call for consensus - adoption of draft-burbridge-lmap-information-model as WG document Thread-Index: AQHO3x4nwl1kPZbVy0C+w4vn4eORgJogftNggAB2rQCAALjYoA== Date: Tue, 12 Nov 2013 15:43:25 +0000 Message-ID: References: <9904FB1B0159DA42B0B887B7FA8119CA1292EAA3@AZ-FFEXMB04.global.avaya.com> <20131111203930.GA770@elstar.local> <20131111214712.GA977@elstar.local> In-Reply-To: <20131111214712.GA977@elstar.local> Accept-Language: en-US Content-Language: en-US X-MS-Has-Attach: X-MS-TNEF-Correlator: x-originating-ip: [151.117.206.7] Content-Type: text/plain; charset="us-ascii" Content-Transfer-Encoding: quoted-printable MIME-Version: 1.0 X-CFilter-Loop: Reflected X-Mailman-Approved-At: Tue, 12 Nov 2013 08:56:38 -0800 Cc: "Cook, Charles" , "'MORTON, ALFRED C \(AL\)'" , "lmap@ietf.org" , 'Trevor Burbridge' , "'Romascanu, Dan \(Dan\)'" , 'KEN KO' , 'marcelo bagnulo braun' , "'Stark, Barbara'" , "'philip.eardley@bt.com'" Subject: Re: [lmap] Comments & opinions on Draft - RE: call for consensus - adoption of draft-burbridge-lmap-information-model as WG document X-BeenThere: lmap@ietf.org X-Mailman-Version: 2.1.12 Precedence: list List-Id: Large Scale Measurement of Access network Performance List-Unsubscribe: , List-Archive: List-Post: List-Help: List-Subscribe: , X-List-Received-Date: Tue, 12 Nov 2013 15:45:03 -0000 Juergon - comments in line > I am still trying to understand what you are suggesting.=20 >=20 > - What does "ISP's need a MA that can Stand by itself as an Overlay" > mean? > - What does "ISP's ... need one that operates more globally as a > universal MA" mean? What is a universal MA? What is more globally > here? > Both of these are answered here -=20 Three large factors on this one - - Opex (operational system cost) Simply put to contain OPEX costs ISP's who also offer VoIP, and Vide= o Over IP like having one test solution. (Brix, Spirent, ....) - Test probe deployment Collision space (if the probes don't know abo= ut each other and don't co-exist well) In general most ISP's have three Op's groups (Broadband, VoIP, a= nd Video over IP) - often each one has it's own system and probes. This occurred due to the timeline of when broadband was a product, = then VoIP (new NOC), then Video (yet another NOC and test system). - NFV future virtualization Virtualization will enable the convergence of items like probes on t= he ISP edge equipment where this testing needs to occur. =20 Summation - A stand alone system to do LMAP is quickly deployable, bu= t drives added cost and Observing the convergence of IP Testing system=20 functionality to contain Broadband testing, VoIP testing, and Video o= ver IP testing vs. One system for each ... It's advisable for LMAP to be = constructed as both a standalone testing overlay, and part of a conve= rged MA to all the functions. I think that's not a problem scope wise provided we recognize it so t= he IPPM group can make tests modular in nature vs. probe specific. > - What registry are you talking about? It seems you are not talking > about the IPPM metrics registry. So what do you have in mind? I'm suggesting we re-align the IPPM metrics registry to fix some of t= he large problems being faced by the industry to ensure the registry succes= s. I think the needs may have changed in the years the registry is being= worked, and feel we should re-assess the functional role the registry play= s. Personally I haven't been able to match a critical need from the ISP,= or Internet user domain with the registry that will make it a wild sucsses= , but I think it's do-able. Thanks, Mike =20 > > Reading again your other email, if you are talking about an MA under co= ntrol of multiple controllers, then I am afraid LMAP 1.0 won't address this= > > (see the WG's charter). But MAs can of course implement tests for mu= ltiple metrics. > > > /js > > --=20 > Juergen Schoenwaelder Jacobs University Bremen gGmbH > Phone: +49 421 200 3587 Campus Ring 1, 28759 Bremen, Germany > Fax: +49 421 200 3103 From steve@idrathernotsay.com Tue Nov 12 10:56:29 2013 Return-Path: X-Original-To: lmap@ietfa.amsl.com Delivered-To: lmap@ietfa.amsl.com Received: from localhost (localhost [127.0.0.1]) by ietfa.amsl.com (Postfix) with ESMTP id 2DD7511E816D for ; Tue, 12 Nov 2013 10:56:29 -0800 (PST) X-Virus-Scanned: amavisd-new at amsl.com X-Spam-Flag: NO X-Spam-Score: -1.996 X-Spam-Level: X-Spam-Status: No, score=-1.996 tagged_above=-999 required=5 tests=[AWL=0.604, BAYES_00=-2.599] Received: from mail.ietf.org ([12.22.58.30]) by localhost (ietfa.amsl.com [127.0.0.1]) (amavisd-new, port 10024) with ESMTP id GozdNCiHSLKP for ; Tue, 12 Nov 2013 10:56:23 -0800 (PST) Received: from colo3.idrathernotsay.com (colo3.idrathernotsay.com [74.207.251.199]) by ietfa.amsl.com (Postfix) with ESMTP id 6AD0911E8167 for ; Tue, 12 Nov 2013 10:56:16 -0800 (PST) Received: from localhost (localhost [127.0.0.1]) by colo3.idrathernotsay.com (Postfix) with ESMTP id 65D6AFC49 for ; Tue, 12 Nov 2013 18:56:15 +0000 (UTC) X-Virus-Scanned: amavisd-new at idrathernotsay.com Received: from colo3.idrathernotsay.com ([127.0.0.1]) by localhost (colo3.idrathernotsay.com [127.0.0.1]) (amavisd-new, port 10024) with ESMTP id v5OXF3EI-9Sf for ; Tue, 12 Nov 2013 18:56:13 +0000 (UTC) Received: from newbie.idrathernotsay.com (pool-173-79-225-215.washdc.fios.verizon.net [173.79.225.215]) by colo3.idrathernotsay.com (Postfix) with ESMTPSA id 77681FC36 for ; Tue, 12 Nov 2013 18:56:13 +0000 (UTC) Received: from newbie.idrathernotsay.com (localhost [127.0.0.1]) by newbie.idrathernotsay.com (Postfix) with ESMTP id 9AC384FBF86 for ; Tue, 12 Nov 2013 13:56:12 -0500 (EST) X-Virus-Scanned: amavisd-new at idrathernotsay.com Received: from newbie.idrathernotsay.com ([127.0.0.1]) by newbie.idrathernotsay.com (newbie.idrathernotsay.com [127.0.0.1]) (amavisd-new, port 10024) with ESMTP id 3BOF84R3QcTS for ; Tue, 12 Nov 2013 13:56:11 -0500 (EST) Received: by newbie.idrathernotsay.com (Postfix, from userid 1000) id C607D4FC11E; Tue, 12 Nov 2013 13:56:11 -0500 (EST) Date: Tue, 12 Nov 2013 13:56:11 -0500 From: Steve Miller To: lmap@ietf.org Message-ID: <20131112185611.GA33370@idrathernotsay.com> References: <20131111182839.GA21454@idrathernotsay.com> <20131111204218.GB770@elstar.local> MIME-Version: 1.0 Content-Type: text/plain; charset=us-ascii Content-Disposition: inline In-Reply-To: <20131111204218.GB770@elstar.local> User-Agent: Mutt/1.5.21 (2010-09-15) Subject: Re: [lmap] Splitting measurement tasks into measurement tasks plus reporting tasks? X-BeenThere: lmap@ietf.org X-Mailman-Version: 2.1.12 Precedence: list List-Id: Large Scale Measurement of Access network Performance List-Unsubscribe: , List-Archive: List-Post: List-Help: List-Subscribe: , X-List-Received-Date: Tue, 12 Nov 2013 18:56:29 -0000 I think that's close but I'm not quite sure it's right. I think part of the gotcha here is that the spec tends to express Measurement Tasks as things that have a definite start and end time, while I'm thinking of things that are running continuously. That is, if we wanted to do a DNS query once a second, forever, the current model would express that more as: Measurement Task "DNS Performance": Perform a DNS loss/latency/jitter check once a second, for 300 seconds, reporting avg/min/max loss, latency and jitter plus a histogram of the latencies. Report Channel "Collector C": Report to collector at 0,5,10,15,20,25,30,35,40,45,50,55 minutes after the hour. Measurement Schedule: Run the Measurement Task "DNS Performance" at 0,5,10,15,20,25,30,35,40,45,50,55 minutes after the hour and report using Report Channel "Collector C". but each of those reports has a very definite start and end time -- there's no way there to define a report via a rolling window. In the current framework, to get what I'm thinking about, I can *almost* do it, but I'd have to do that separately. First, to get the performance numbers every five minutes, we'd have what's above, more or less: Measurement Task "DNS Performance": Perform a DNS loss/latency/jitter check once a second, for 300 seconds, reporting avg/min/max loss, latency and jitter plus a histogram of the latencies. Report Channel "Collector C 300Secs": Report to collector at 0,5,10,15,20,25,30,35,40,45,50,55 minutes after the hour. Measurement Schedule: Run the Measurement Task "DNS Performance" at 0,5,10,15,20,25,30,35,40,45,50,55 minutes after the hour and report using Report Channel "Collector C 300Secs". and then to get some sort of report immediately, it seems like about as close as we'd get would be: Measurement Task "DNS Badness": Perform a DNS loss/latency/jitter check once a second, for 300 seconds, reporting avg/min/max loss, latency and jitter immediately if the loss is worse than 1%, the latency is more than 200ms, or the jitter is more than 200ms. Report Channel "Collector C 1Sec": Report to collector once a second. Measurement Schedule: Run the Measurement Task "DNS Badness" at 0,5,10,15,20,25,30,35,40,45,50,55 minutes after the hour and report using Report Channel "Collector C 1Sec". which has a couple of issues: - now we're sending twice as much traffic to each destination as we were before (which in this example isn't such a big deal but it's not hard to imagine that a file-download test or an RTP test wouldn't be so forgiving (-: ) - it's not truly a rolling window, so you either have to clamp the accelerated reporting until you have enough baseline to be useful (that is, you don't want to report "OMG! 100% loss!" if just the first packet is lost) or you have to put up with noisy reports - I think the collector gets a report every second, even if that report is empty Maybe a way to look at this would be to have the concepts of: - never-ending measurements, which run continuously and - async report channels, where you could express simple conditions to control whether or not a report occurs or maybe async report channels are really some sort of async measurement schedule: Run the test named "DNS Badness" continuously; report using the channel "Collector C" whenever loss is more than 1% or the latency is more than 200ms or the jitter is more than 200ms over the last 300 seconds. I could be convinced that I'm thinking too hard about this or that my perception of the measurement model as viewing Measurement Tasks as discrete things with discrete start times and durations is simply incorrect. Did that make anything more clear? Also, to address some of what you and Trevor were discussing: I don't know that I necessarily regard these async reports as logging or status. I think they're closer to "normal" reports, they just happen asynchronously. I could be convinced to treat this as logging or status, too, if that'd be helpful somehow. -Steve On Mon, Nov 11, 2013 at 09:42:18PM +0100, Juergen Schoenwaelder wrote: > I hear two suggestions here: > > a) Give reporting channels their own independent schedule (right now > they are essentially tied to the scheduled task). > > b) Provide a means for post-processing raw data in order to derive > certain statistics before the result is shipped via a report > channel. > > Is this a fair summary of the request? From j.schoenwaelder@jacobs-university.de Tue Nov 12 11:26:10 2013 Return-Path: X-Original-To: lmap@ietfa.amsl.com Delivered-To: lmap@ietfa.amsl.com Received: from localhost (localhost [127.0.0.1]) by ietfa.amsl.com (Postfix) with ESMTP id 721DA21E80AC for ; Tue, 12 Nov 2013 11:26:10 -0800 (PST) X-Virus-Scanned: amavisd-new at amsl.com X-Spam-Flag: NO X-Spam-Score: -103.205 X-Spam-Level: X-Spam-Status: No, score=-103.205 tagged_above=-999 required=5 tests=[AWL=0.044, BAYES_00=-2.599, HELO_EQ_DE=0.35, RCVD_IN_DNSWL_LOW=-1, USER_IN_WHITELIST=-100] Received: from mail.ietf.org ([12.22.58.30]) by localhost (ietfa.amsl.com [127.0.0.1]) (amavisd-new, port 10024) with ESMTP id sF+eM-ejc74s for ; Tue, 12 Nov 2013 11:26:05 -0800 (PST) Received: from hermes.jacobs-university.de (hermes.jacobs-university.de [212.201.44.23]) by ietfa.amsl.com (Postfix) with ESMTP id A65AD21E80A1 for ; Tue, 12 Nov 2013 11:26:03 -0800 (PST) Received: from localhost (demetrius1.jacobs-university.de [212.201.44.46]) by hermes.jacobs-university.de (Postfix) with ESMTP id 3A16920047; Tue, 12 Nov 2013 20:26:02 +0100 (CET) X-Virus-Scanned: amavisd-new at jacobs-university.de Received: from hermes.jacobs-university.de ([212.201.44.23]) by localhost (demetrius1.jacobs-university.de [212.201.44.32]) (amavisd-new, port 10024) with ESMTP id dpfBg5GwN4d6; Tue, 12 Nov 2013 20:26:02 +0100 (CET) Received: from elstar.local (elstar.jacobs.jacobs-university.de [10.50.231.133]) by hermes.jacobs-university.de (Postfix) with ESMTP id 25AC920041; Tue, 12 Nov 2013 20:26:01 +0100 (CET) Received: by elstar.local (Postfix, from userid 501) id B5D8729467DD; Tue, 12 Nov 2013 20:25:55 +0100 (CET) Date: Tue, 12 Nov 2013 20:25:55 +0100 From: Juergen Schoenwaelder To: Steve Miller Message-ID: <20131112192555.GB7537@elstar.local> Mail-Followup-To: Steve Miller , lmap@ietf.org References: <20131111182839.GA21454@idrathernotsay.com> <20131111204218.GB770@elstar.local> <20131112185611.GA33370@idrathernotsay.com> MIME-Version: 1.0 Content-Type: text/plain; charset=us-ascii Content-Disposition: inline In-Reply-To: <20131112185611.GA33370@idrathernotsay.com> User-Agent: Mutt/1.5.21 (2010-09-15) Cc: lmap@ietf.org Subject: Re: [lmap] Splitting measurement tasks into measurement tasks plus reporting tasks? X-BeenThere: lmap@ietf.org X-Mailman-Version: 2.1.12 Precedence: list Reply-To: Juergen Schoenwaelder List-Id: Large Scale Measurement of Access network Performance List-Unsubscribe: , List-Archive: List-Post: List-Help: List-Subscribe: , X-List-Received-Date: Tue, 12 Nov 2013 19:26:10 -0000 I think you touch on several issues: a) Are reports send if nothing is pending to be reported? I think the answer should be no but this can be discussed. b) Should we support a notion of filters that can be applied, that is only result records passing a filter expression are actually reported. c) In a previous email, someone suggested to have aggregation functions that can be applied generically to test results (e.g., calculation of avg, min, max, ...). d) Should we support a report timing that kind of says "report immediately if a result record is available" These are all fair questions one can ask and it is going to be a trade-off between simplicity and functional richness. Perhaps we need to look at what deployed systems really do so we do not get over board building aggregation and filtering features into the model that can also easily be applied on the collector's end. /js On Tue, Nov 12, 2013 at 01:56:11PM -0500, Steve Miller wrote: > I think that's close but I'm not quite sure it's right. I think part of the gotcha here is that the spec tends to express Measurement Tasks as things that have a definite start and end time, while I'm thinking of things that are running continuously. > > That is, if we wanted to do a DNS query once a second, forever, the current model would express that more as: > > Measurement Task "DNS Performance": Perform a DNS loss/latency/jitter check once a second, for 300 seconds, reporting avg/min/max loss, latency and jitter plus a histogram of the latencies. > > Report Channel "Collector C": Report to collector at 0,5,10,15,20,25,30,35,40,45,50,55 minutes after the hour. > > Measurement Schedule: Run the Measurement Task "DNS Performance" at 0,5,10,15,20,25,30,35,40,45,50,55 minutes after the hour and report using Report Channel "Collector C". > > but each of those reports has a very definite start and end time -- there's no way there to define a report via a rolling window. > > In the current framework, to get what I'm thinking about, I can *almost* do it, but I'd have to do that separately. First, to get the performance numbers every five minutes, we'd have what's above, more or less: > > Measurement Task "DNS Performance": Perform a DNS loss/latency/jitter check once a second, for 300 seconds, reporting avg/min/max loss, latency and jitter plus a histogram of the latencies. > > Report Channel "Collector C 300Secs": Report to collector at 0,5,10,15,20,25,30,35,40,45,50,55 minutes after the hour. > > Measurement Schedule: Run the Measurement Task "DNS Performance" at 0,5,10,15,20,25,30,35,40,45,50,55 minutes after the hour and report using Report Channel "Collector C 300Secs". > > and then to get some sort of report immediately, it seems like about as close as we'd get would be: > > Measurement Task "DNS Badness": Perform a DNS loss/latency/jitter check once a second, for 300 seconds, reporting avg/min/max loss, latency and jitter immediately if the loss is worse than 1%, the latency is more than 200ms, or the jitter is more than 200ms. > > Report Channel "Collector C 1Sec": Report to collector once a second. > > Measurement Schedule: Run the Measurement Task "DNS Badness" at 0,5,10,15,20,25,30,35,40,45,50,55 minutes after the hour and report using Report Channel "Collector C 1Sec". > > which has a couple of issues: > > - now we're sending twice as much traffic to each destination as we were before (which in this example isn't such a big deal but it's not hard to imagine that a file-download test or an RTP test wouldn't be so forgiving (-: ) > > - it's not truly a rolling window, so you either have to clamp the accelerated reporting until you have enough baseline to be useful (that is, you don't want to report "OMG! 100% loss!" if just the first packet is lost) or you have to put up with noisy reports > > - I think the collector gets a report every second, even if that report is empty > > Maybe a way to look at this would be to have the concepts of: > > - never-ending measurements, which run continuously > > and > > - async report channels, where you could express simple conditions to control whether or not a report occurs > > or maybe async report channels are really some sort of async measurement schedule: > > Run the test named "DNS Badness" continuously; report using the channel "Collector C" whenever loss is more than 1% or the latency is more than 200ms or the jitter is more than 200ms over the last 300 seconds. > > I could be convinced that I'm thinking too hard about this or that my perception of the measurement model as viewing Measurement Tasks as discrete things with discrete start times and durations is simply incorrect. > > Did that make anything more clear? > > Also, to address some of what you and Trevor were discussing: I don't know that I necessarily regard these async reports as logging or status. I think they're closer to "normal" reports, they just happen asynchronously. I could be convinced to treat this as logging or status, too, if that'd be helpful somehow. > > -Steve > > On Mon, Nov 11, 2013 at 09:42:18PM +0100, Juergen Schoenwaelder wrote: > > I hear two suggestions here: > > > > a) Give reporting channels their own independent schedule (right now > > they are essentially tied to the scheduled task). > > > > b) Provide a means for post-processing raw data in order to derive > > certain statistics before the result is shipped via a report > > channel. > > > > Is this a fair summary of the request? > _______________________________________________ > lmap mailing list > lmap@ietf.org > https://www.ietf.org/mailman/listinfo/lmap -- Juergen Schoenwaelder Jacobs University Bremen gGmbH Phone: +49 421 200 3587 Campus Ring 1, 28759 Bremen, Germany Fax: +49 421 200 3103 From j.schoenwaelder@jacobs-university.de Tue Nov 12 11:11:28 2013 Return-Path: X-Original-To: lmap@ietfa.amsl.com Delivered-To: lmap@ietfa.amsl.com Received: from localhost (localhost [127.0.0.1]) by ietfa.amsl.com (Postfix) with ESMTP id 4B96B21E8064 for ; Tue, 12 Nov 2013 11:11:28 -0800 (PST) X-Virus-Scanned: amavisd-new at amsl.com X-Spam-Flag: NO X-Spam-Score: -103.205 X-Spam-Level: X-Spam-Status: No, score=-103.205 tagged_above=-999 required=5 tests=[AWL=0.044, BAYES_00=-2.599, HELO_EQ_DE=0.35, RCVD_IN_DNSWL_LOW=-1, USER_IN_WHITELIST=-100] Received: from mail.ietf.org ([12.22.58.30]) by localhost (ietfa.amsl.com [127.0.0.1]) (amavisd-new, port 10024) with ESMTP id gVKwczgU6iX8 for ; Tue, 12 Nov 2013 11:11:14 -0800 (PST) Received: from hermes.jacobs-university.de (hermes.jacobs-university.de [212.201.44.23]) by ietfa.amsl.com (Postfix) with ESMTP id E7D6621F9E39 for ; Tue, 12 Nov 2013 11:11:13 -0800 (PST) Received: from localhost (demetrius4.jacobs-university.de [212.201.44.49]) by hermes.jacobs-university.de (Postfix) with ESMTP id 5C80A20047; Tue, 12 Nov 2013 20:11:13 +0100 (CET) X-Virus-Scanned: amavisd-new at jacobs-university.de Received: from hermes.jacobs-university.de ([212.201.44.23]) by localhost (demetrius4.jacobs-university.de [212.201.44.32]) (amavisd-new, port 10024) with ESMTP id 9o7IHYZR1eYx; Tue, 12 Nov 2013 20:11:13 +0100 (CET) Received: from elstar.local (elstar.jacobs.jacobs-university.de [10.50.231.133]) by hermes.jacobs-university.de (Postfix) with ESMTP id 3864020041; Tue, 12 Nov 2013 20:11:12 +0100 (CET) Received: by elstar.local (Postfix, from userid 501) id 9753D29466D0; Tue, 12 Nov 2013 20:11:06 +0100 (CET) Date: Tue, 12 Nov 2013 20:11:06 +0100 From: Juergen Schoenwaelder To: "Bugenhagen, Michael K" Message-ID: <20131112191106.GA7537@elstar.local> Mail-Followup-To: "Bugenhagen, Michael K" , "Cook, Charles" , "'MORTON, ALFRED C (AL)'" , "lmap@ietf.org" , 'Trevor Burbridge' , "'Romascanu, Dan (Dan)'" , 'KEN KO' , 'marcelo bagnulo braun' , "'Stark, Barbara'" , "'philip.eardley@bt.com'" References: <9904FB1B0159DA42B0B887B7FA8119CA1292EAA3@AZ-FFEXMB04.global.avaya.com> <20131111203930.GA770@elstar.local> <20131111214712.GA977@elstar.local> MIME-Version: 1.0 Content-Type: text/plain; charset=us-ascii Content-Disposition: inline In-Reply-To: User-Agent: Mutt/1.5.21 (2010-09-15) X-Mailman-Approved-At: Tue, 12 Nov 2013 12:37:02 -0800 Cc: "Cook, Charles" , "'MORTON, ALFRED C \(AL\)'" , "lmap@ietf.org" , 'Trevor Burbridge' , "'Romascanu, Dan \(Dan\)'" , 'KEN KO' , 'marcelo bagnulo braun' , "'Stark, Barbara'" , "'philip.eardley@bt.com'" Subject: Re: [lmap] Comments & opinions on Draft - RE: call for consensus - adoption of draft-burbridge-lmap-information-model as WG document X-BeenThere: lmap@ietf.org X-Mailman-Version: 2.1.12 Precedence: list Reply-To: Juergen Schoenwaelder List-Id: Large Scale Measurement of Access network Performance List-Unsubscribe: , List-Archive: List-Post: List-Help: List-Subscribe: , X-List-Received-Date: Tue, 12 Nov 2013 19:11:28 -0000 On Tue, Nov 12, 2013 at 03:43:25PM +0000, Bugenhagen, Michael K wrote: > Juergon - comments in line > > > > I am still trying to understand what you are suggesting. > > > > - What does "ISP's need a MA that can Stand by itself as an Overlay" > > mean? > > - What does "ISP's ... need one that operates more globally as a > > universal MA" mean? What is a universal MA? What is more globally > > here? > > > Both of these are answered here - > > Three large factors on this one - > - Opex (operational system cost) > Simply put to contain OPEX costs ISP's who also offer VoIP, and Video Over IP like having one test solution. (Brix, Spirent, ....) > - Test probe deployment Collision space (if the probes don't know about each other and don't co-exist well) > In general most ISP's have three Op's groups (Broadband, VoIP, and Video over IP) - often each one has it's own system and probes. > This occurred due to the timeline of when broadband was a product, then VoIP (new NOC), then Video (yet another NOC and test system). > - NFV future virtualization > Virtualization will enable the convergence of items like probes on the ISP edge equipment where this testing needs to occur. > > Summation - A stand alone system to do LMAP is quickly deployable, but drives added cost and Observing the convergence of IP Testing system > functionality to contain Broadband testing, VoIP testing, and Video over IP testing vs. One system for each ... It's advisable for LMAP to be constructed as both a standalone testing overlay, and part of a converged MA to all the functions. > I think that's not a problem scope wise provided we recognize it so the IPPM group can make tests modular in nature vs. probe specific. > I still do not understand very well. If your concern is that MAs are separate devices that I can tell you that this concern is not justified. An MA is a function, how it is packaged into devices is left to implementors. > > - What registry are you talking about? It seems you are not talking > > about the IPPM metrics registry. So what do you have in mind? > > I'm suggesting we re-align the IPPM metrics registry to fix some of the large problems being faced by the industry to ensure the registry success. > I think the needs may have changed in the years the registry is being worked, and feel we should re-assess the functional role the registry plays. > Personally I haven't been able to match a critical need from the ISP, or Internet user domain with the registry that will make it a wild sucsses, but I think it's do-able. I assume IPPM folks can make sense out of your request. /js -- Juergen Schoenwaelder Jacobs University Bremen gGmbH Phone: +49 421 200 3587 Campus Ring 1, 28759 Bremen, Germany Fax: +49 421 200 3103 From jason.weil@twcable.com Tue Nov 12 14:12:11 2013 Return-Path: X-Original-To: lmap@ietfa.amsl.com Delivered-To: lmap@ietfa.amsl.com Received: from localhost (localhost [127.0.0.1]) by ietfa.amsl.com (Postfix) with ESMTP id 19D6D21E813C for ; Tue, 12 Nov 2013 14:12:11 -0800 (PST) X-Virus-Scanned: amavisd-new at amsl.com X-Spam-Flag: NO X-Spam-Score: -0.425 X-Spam-Level: X-Spam-Status: No, score=-0.425 tagged_above=-999 required=5 tests=[AWL=0.038, BAYES_00=-2.599, HELO_EQ_MODEMCABLE=0.768, HOST_EQ_MODEMCABLE=1.368] Received: from mail.ietf.org ([12.22.58.30]) by localhost (ietfa.amsl.com [127.0.0.1]) (amavisd-new, port 10024) with ESMTP id tofgjEJANNPD for ; Tue, 12 Nov 2013 14:12:07 -0800 (PST) Received: from cdcipgw02.twcable.com (cdcipgw02.twcable.com [165.237.91.111]) by ietfa.amsl.com (Postfix) with ESMTP id CF87A21E80E4 for ; Tue, 12 Nov 2013 14:12:06 -0800 (PST) X-SENDER-IP: 10.136.163.15 X-SENDER-REPUTATION: None X-IronPort-AV: E=Sophos;i="4.93,688,1378872000"; d="scan'208";a="48214383" Received: from unknown (HELO PRVPEXHUB06.corp.twcable.com) ([10.136.163.15]) by cdcipgw02.twcable.com with ESMTP/TLS/RC4-MD5; 12 Nov 2013 17:11:16 -0500 Received: from PRVPEXVS17.corp.twcable.com ([10.136.163.95]) by PRVPEXHUB06.corp.twcable.com ([10.136.163.15]) with mapi; Tue, 12 Nov 2013 17:11:58 -0500 From: "Weil, Jason" To: Juergen Schoenwaelder , "Bugenhagen, Michael K" Date: Tue, 12 Nov 2013 17:11:55 -0500 Thread-Topic: [lmap] Comments & opinions on Draft - RE: call for consensus - adoption of draft-burbridge-lmap-information-model as WG document Thread-Index: Ac7f9DOYkitTBtXkS7+7cDmKkY9kZg== Message-ID: In-Reply-To: <20131112191106.GA7537@elstar.local> Accept-Language: en-US Content-Language: en-US X-MS-Has-Attach: X-MS-TNEF-Correlator: user-agent: Microsoft-MacOutlook/14.3.6.130613 acceptlanguage: en-US Content-Type: text/plain; charset="us-ascii" Content-Transfer-Encoding: quoted-printable MIME-Version: 1.0 X-Mailman-Approved-At: Tue, 12 Nov 2013 14:17:40 -0800 Cc: "Cook, Charles" , "'MORTON, ALFRED C \(AL\)'" , "lmap@ietf.org" , 'Trevor Burbridge' , "'Romascanu, Dan \(Dan\)'" , 'KEN KO' , 'marcelo bagnulo braun' , "'Stark, Barbara'" , "'philip.eardley@bt.com'" Subject: Re: [lmap] Comments & opinions on Draft - RE: call for consensus - adoption of draft-burbridge-lmap-information-model as WG document X-BeenThere: lmap@ietf.org X-Mailman-Version: 2.1.12 Precedence: list List-Id: Large Scale Measurement of Access network Performance List-Unsubscribe: , List-Archive: List-Post: List-Help: List-Subscribe: , X-List-Received-Date: Tue, 12 Nov 2013 22:12:11 -0000 I am wondering if we may be discussing two different issues here. It feels like part of this discussion is about the MA Function in the LMAP Framework and another part of this discussion is concerning a hardware device housing one or more MA Functions. An ISP speccing a device through the BBF may/should be concerned with the question of how multiple MAs under the same org and management system will interact with each other. It might be useful to reiterate some language from the charter regarding our current scope as well: 1. No requirement for coordination of overlapping LMAP domains: "It is assumed that different organization's LMAP measurement domains can overlap, and that active measurement packets appear along with normal user traffic when crossing another organization's network. There is no requirement to specify a mechanism for coordination between the LMAP measurements in overlapping domains (for instance a home network with MAs on the home gateway, set top box and laptop). In principle, there are no restrictions on the type of device in which the MA function resides." 2. Service parameter discovery and sharing is out of scope "Discovering the service parameters on the MAs or sharing the service parameters between MAs are out of the scope." 3. LMAP should coordinate with other SDOs on the Information model. "The LMAP working group will coordinate with other standards bodies working in this area (e.g., BBF, IEEE 802.16, ETSI) regarding the information model, and with other IETF working groups in the areas of data models, protocols, multiple interface management, and measurement of performance metrics. Thanks, Jason On 11/12/13 2:11 PM, "Juergen Schoenwaelder" wrote: >On Tue, Nov 12, 2013 at 03:43:25PM +0000, Bugenhagen, Michael K wrote: >> Juergon - comments in line >> >> >> > I am still trying to understand what you are suggesting. >> > >> > - What does "ISP's need a MA that can Stand by itself as an Overlay" >> > mean? >> > - What does "ISP's ... need one that operates more globally as a >> > universal MA" mean? What is a universal MA? What is more globally >> > here? >> > >> Both of these are answered here - >> >> Three large factors on this one - >> - Opex (operational system cost) >> Simply put to contain OPEX costs ISP's who also off= er VoIP, and >>Video Over IP like having one test solution. (Brix, Spirent, ....) >> - Test probe deployment Collision space (if the probes don'= t know >>about each other and don't co-exist well) >> In general most ISP's have three Op's groups (Broad= band, >>VoIP, and Video over IP) - often each one has it's own system and probes. >> This occurred due to the timeline of when broadban= d was a >>product, then VoIP (new NOC), then Video (yet another NOC and test >>system). >> - NFV future virtualization >> Virtualization will enable the convergence of items= like probes >>on the ISP edge equipment where this testing needs to occur. >> >> Summation - A stand alone system to do LMAP is quickly >>deployable, but drives added cost and Observing the convergence of IP >>Testing system >> functionality to contain Broadband testing, VoIP testing, and >>Video over IP testing vs. One system for each ... It's advisable for >>LMAP to be constructed as both a standalone testing overlay, and >>part of a converged MA to all the functions. >> I think that's not a problem scope wise provided we recogni= ze it >>so the IPPM group can make tests modular in nature vs. probe specific. >> > >I still do not understand very well. If your concern is that MAs are >separate devices that I can tell you that this concern is not >justified. An MA is a function, how it is packaged into devices is >left to implementors. > >> > - What registry are you talking about? It seems you are not talking >> > about the IPPM metrics registry. So what do you have in mind? >> >> I'm suggesting we re-align the IPPM metrics registry to fix some >>of the large problems being faced by the industry to ensure the registry >>success. >> I think the needs may have changed in the years the registry is >>being worked, and feel we should re-assess the functional role the >>registry plays. >> Personally I haven't been able to match a critical need from the >>ISP, or Internet user domain with the registry that will make it a wild >>sucsses, but I think it's do-able. > >I assume IPPM folks can make sense out of your request. > >/js > >-- >Juergen Schoenwaelder Jacobs University Bremen gGmbH >Phone: +49 421 200 3587 Campus Ring 1, 28759 Bremen, Germany >Fax: +49 421 200 3103 >_______________________________________________ >lmap mailing list >lmap@ietf.org >https://www.ietf.org/mailman/listinfo/lmap This E-mail and any of its attachments may contain Time Warner Cable propri= etary information, which is privileged, confidential, or subject to copyrig= ht belonging to Time Warner Cable. This E-mail is intended solely for the u= se of the individual or entity to which it is addressed. If you are not the= intended recipient of this E-mail, you are hereby notified that any dissem= ination, distribution, copying, or action taken in relation to the contents= of and attachments to this E-mail is strictly prohibited and may be unlawf= ul. If you have received this E-mail in error, please notify the sender imm= ediately and permanently delete the original and any copy of this E-mail an= d any printout. From trevor.burbridge@bt.com Wed Nov 13 00:53:49 2013 Return-Path: X-Original-To: lmap@ietfa.amsl.com Delivered-To: lmap@ietfa.amsl.com Received: from localhost (localhost [127.0.0.1]) by ietfa.amsl.com (Postfix) with ESMTP id AC13421E8082 for ; Wed, 13 Nov 2013 00:53:49 -0800 (PST) X-Virus-Scanned: amavisd-new at amsl.com X-Spam-Flag: NO X-Spam-Score: -3.239 X-Spam-Level: X-Spam-Status: No, score=-3.239 tagged_above=-999 required=5 tests=[AWL=0.360, BAYES_00=-2.599, RCVD_IN_DNSWL_LOW=-1] Received: from mail.ietf.org ([12.22.58.30]) by localhost (ietfa.amsl.com [127.0.0.1]) (amavisd-new, port 10024) with ESMTP id g2Kf49vrfXP8 for ; Wed, 13 Nov 2013 00:53:45 -0800 (PST) Received: from smtpe1.intersmtp.com (smtp64.intersmtp.com [62.239.224.237]) by ietfa.amsl.com (Postfix) with ESMTP id 8EBB721E808D for ; Wed, 13 Nov 2013 00:53:44 -0800 (PST) Received: from EVMHT62-UKRD.domain1.systemhost.net (10.36.3.128) by RDW083A008ED64.smtp-e4.hygiene.service (10.187.98.13) with Microsoft SMTP Server (TLS) id 8.3.298.1; Wed, 13 Nov 2013 08:53:43 +0000 Received: from EMV64-UKRD.domain1.systemhost.net ([169.254.2.72]) by EVMHT62-UKRD.domain1.systemhost.net ([10.36.3.128]) with mapi; Wed, 13 Nov 2013 08:53:43 +0000 From: To: , Date: Wed, 13 Nov 2013 08:53:41 +0000 Thread-Topic: [lmap] Splitting measurement tasks into measurement tasks plus reporting tasks? Thread-Index: Ac7f3RHPVNm6QgHjRBOnAGHL38l56AAcAT5g Message-ID: References: <20131111182839.GA21454@idrathernotsay.com> <20131111204218.GB770@elstar.local> <20131112185611.GA33370@idrathernotsay.com> <20131112192555.GB7537@elstar.local> In-Reply-To: <20131112192555.GB7537@elstar.local> Accept-Language: en-US, en-GB Content-Language: en-US X-MS-Has-Attach: X-MS-TNEF-Correlator: acceptlanguage: en-US, en-GB Content-Type: text/plain; charset="us-ascii" Content-Transfer-Encoding: quoted-printable MIME-Version: 1.0 Cc: lmap@ietf.org Subject: Re: [lmap] Splitting measurement tasks into measurement tasks plus reporting tasks? X-BeenThere: lmap@ietf.org X-Mailman-Version: 2.1.12 Precedence: list List-Id: Large Scale Measurement of Access network Performance List-Unsubscribe: , List-Archive: List-Post: List-Help: List-Subscribe: , X-List-Received-Date: Wed, 13 Nov 2013 08:53:49 -0000 (d) we have in the information model. (a)-(c) are interesting questions. (a) could always be a configurable param= eter. (b) and (c) I've kept away from for reasons of complexity, but they c= ould be useful tools. My initial reaction is that we don't need pre-process= ing as (i) the amount of data returned is not likely to be huge in most sce= narios; (ii) some of this can be done within the test - for example a strea= m of ping packets can be a single measurement and report avg, min, max etc. Trevor. >-----Original Message----- >From: lmap-bounces@ietf.org [mailto:lmap-bounces@ietf.org] On Behalf Of >Juergen Schoenwaelder >Sent: 12 November 2013 19:26 >To: Steve Miller >Cc: lmap@ietf.org >Subject: Re: [lmap] Splitting measurement tasks into measurement tasks plu= s >reporting tasks? > >I think you touch on several issues: > >a) Are reports send if nothing is pending to be reported? > > I think the answer should be no but this can be discussed. > >b) Should we support a notion of filters that can be applied, that is > only result records passing a filter expression are actually reported. > >c) In a previous email, someone suggested to have aggregation > functions that can be applied generically to test results > (e.g., calculation of avg, min, max, ...). > >d) Should we support a report timing that kind of says "report > immediately if a result record is available" > >These are all fair questions one can ask and it is going to be a >trade-off between simplicity and functional richness. Perhaps we need >to look at what deployed systems really do so we do not get over board >building aggregation and filtering features into the model that can >also easily be applied on the collector's end. > >/js > >On Tue, Nov 12, 2013 at 01:56:11PM -0500, Steve Miller wrote: >> I think that's close but I'm not quite sure it's right. I think part= of the gotcha >here is that the spec tends to express Measurement Tasks as things that ha= ve a >definite start and end time, while I'm thinking of things that are running >continuously. >> >> That is, if we wanted to do a DNS query once a second, forever, the c= urrent >model would express that more as: >> >> Measurement Task "DNS Performance": Perform a DNS >loss/latency/jitter check once a second, for 300 seconds, reporting avg/mi= n/max >loss, latency and jitter plus a histogram of the latencies. >> >> Report Channel "Collector C": Report to collector at >0,5,10,15,20,25,30,35,40,45,50,55 minutes after the hour. >> >> Measurement Schedule: Run the Measurement Task "DNS Performance" >at 0,5,10,15,20,25,30,35,40,45,50,55 minutes after the hour and report usi= ng >Report Channel "Collector C". >> >> but each of those reports has a very definite start and end time -- ther= e's no >way there to define a report via a rolling window. >> >> In the current framework, to get what I'm thinking about, I can *almo= st* do >it, but I'd have to do that separately. First, to get the performance num= bers every >five minutes, we'd have what's above, more or less: >> >> Measurement Task "DNS Performance": Perform a DNS >loss/latency/jitter check once a second, for 300 seconds, reporting avg/mi= n/max >loss, latency and jitter plus a histogram of the latencies. >> >> Report Channel "Collector C 300Secs": Report to collector at >0,5,10,15,20,25,30,35,40,45,50,55 minutes after the hour. >> >> Measurement Schedule: Run the Measurement Task "DNS Performance" >at 0,5,10,15,20,25,30,35,40,45,50,55 minutes after the hour and report usi= ng >Report Channel "Collector C 300Secs". >> >> and then to get some sort of report immediately, it seems like about as = close as >we'd get would be: >> >> Measurement Task "DNS Badness": Perform a DNS loss/latency/jitter >check once a second, for 300 seconds, reporting avg/min/max loss, latency = and >jitter immediately if the loss is worse than 1%, the latency is more than = 200ms, or >the jitter is more than 200ms. >> >> Report Channel "Collector C 1Sec": Report to collector once a second. >> >> Measurement Schedule: Run the Measurement Task "DNS Badness" at >0,5,10,15,20,25,30,35,40,45,50,55 minutes after the hour and report using >Report Channel "Collector C 1Sec". >> >> which has a couple of issues: >> >> - now we're sending twice as much traffic to each destination as we wer= e >before (which in this example isn't such a big deal but it's not hard to i= magine that >a file-download test or an RTP test wouldn't be so forgiving (-: ) >> >> - it's not truly a rolling window, so you either have to clamp the >accelerated reporting until you have enough baseline to be useful (that is= , you >don't want to report "OMG! 100% loss!" if just the first packet is lost) = or you have >to put up with noisy reports >> >> - I think the collector gets a report every second, even if that report= is >empty >> >> Maybe a way to look at this would be to have the concepts of: >> >> - never-ending measurements, which run continuously >> >> and >> >> - async report channels, where you could express simple conditions to >control whether or not a report occurs >> >> or maybe async report channels are really some sort of async measurement >schedule: >> >> Run the test named "DNS Badness" continuously; report using the >channel "Collector C" whenever loss is more than 1% or the latency is more= than >200ms or the jitter is more than 200ms over the last 300 seconds. >> >> I could be convinced that I'm thinking too hard about this or that my >perception of the measurement model as viewing Measurement Tasks as discre= te >things with discrete start times and durations is simply incorrect. >> >> Did that make anything more clear? >> >> Also, to address some of what you and Trevor were discussing: I don't= know >that I necessarily regard these async reports as logging or status. I thi= nk they're >closer to "normal" reports, they just happen asynchronously. I could be >convinced to treat this as logging or status, too, if that'd be helpful so= mehow. >> >> -Steve >> >> On Mon, Nov 11, 2013 at 09:42:18PM +0100, Juergen Schoenwaelder wrote: >> > I hear two suggestions here: >> > >> > a) Give reporting channels their own independent schedule (right now >> > they are essentially tied to the scheduled task). >> > >> > b) Provide a means for post-processing raw data in order to derive >> > certain statistics before the result is shipped via a report >> > channel. >> > >> > Is this a fair summary of the request? >> _______________________________________________ >> lmap mailing list >> lmap@ietf.org >> https://www.ietf.org/mailman/listinfo/lmap > >-- >Juergen Schoenwaelder Jacobs University Bremen gGmbH >Phone: +49 421 200 3587 Campus Ring 1, 28759 Bremen, Germany >Fax: +49 421 200 3103 >_______________________________________________ >lmap mailing list >lmap@ietf.org >https://www.ietf.org/mailman/listinfo/lmap From acmorton@att.com Wed Nov 13 06:41:11 2013 Return-Path: X-Original-To: lmap@ietfa.amsl.com Delivered-To: lmap@ietfa.amsl.com Received: from localhost (localhost [127.0.0.1]) by ietfa.amsl.com (Postfix) with ESMTP id 553CC11E815A for ; Wed, 13 Nov 2013 06:41:11 -0800 (PST) X-Virus-Scanned: amavisd-new at amsl.com X-Spam-Flag: NO X-Spam-Score: -105.742 X-Spam-Level: X-Spam-Status: No, score=-105.742 tagged_above=-999 required=5 tests=[AWL=0.857, BAYES_00=-2.599, RCVD_IN_DNSWL_MED=-4, USER_IN_WHITELIST=-100] Received: from mail.ietf.org ([12.22.58.30]) by localhost (ietfa.amsl.com [127.0.0.1]) (amavisd-new, port 10024) with ESMTP id VQFpfg7FYQ85 for ; Wed, 13 Nov 2013 06:41:04 -0800 (PST) Received: from nbfkord-smmo06.seg.att.com (nbfkord-smmo06.seg.att.com [209.65.160.94]) by ietfa.amsl.com (Postfix) with ESMTP id 6594511E8180 for ; Wed, 13 Nov 2013 06:41:04 -0800 (PST) Received: from unknown [144.160.229.24] (EHLO nbfkord-smmo06.seg.att.com) by nbfkord-smmo06.seg.att.com(mxl_mta-6.15.0-1) with ESMTP id 08f83825.67769940.211502.00-550.586598.nbfkord-smmo06.seg.att.com (envelope-from ); Wed, 13 Nov 2013 14:41:04 +0000 (UTC) X-MXL-Hash: 52838f8037ee6b63-aeab3ba231fc5c0c94631caaa49e9fb2df765d25 Received: from unknown [144.160.229.24] (EHLO alpi155.enaf.aldc.att.com) by nbfkord-smmo06.seg.att.com(mxl_mta-6.15.0-1) over TLS secured channel with ESMTP id a7f83825.0.211462.00-184.586466.nbfkord-smmo06.seg.att.com (envelope-from ); Wed, 13 Nov 2013 14:40:59 +0000 (UTC) X-MXL-Hash: 52838f7b5084f08e-74ec0242c8d8c20f957af51878fab14e895ba54c Received: from enaf.aldc.att.com (localhost [127.0.0.1]) by alpi155.enaf.aldc.att.com (8.14.5/8.14.5) with ESMTP id rADEevtk027256; Wed, 13 Nov 2013 09:40:58 -0500 Received: from mlpi408.sfdc.sbc.com (mlpi408.sfdc.sbc.com [130.9.128.240]) by alpi155.enaf.aldc.att.com (8.14.5/8.14.5) with ESMTP id rADEelV6027118 (version=TLSv1/SSLv3 cipher=DHE-RSA-AES256-SHA bits=256 verify=NO); Wed, 13 Nov 2013 09:40:48 -0500 Received: from clpi183.sldc.sbc.com (clpi183.sldc.sbc.com [135.41.1.46]) by mlpi408.sfdc.sbc.com (RSA Interceptor); Wed, 13 Nov 2013 14:40:34 GMT Received: from sldc.sbc.com (localhost [127.0.0.1]) by clpi183.sldc.sbc.com (8.14.5/8.14.5) with ESMTP id rADEeXfU029144; Wed, 13 Nov 2013 08:40:33 -0600 Received: from mail-azure.research.att.com (mail-azure.research.att.com [135.207.255.18]) by clpi183.sldc.sbc.com (8.14.5/8.14.5) with ESMTP id rADEeQAM028570; Wed, 13 Nov 2013 08:40:26 -0600 Received: from njfpsrvexg2.research.att.com (njfpsrvexg2.research.att.com [135.207.160.21]) by mail-azure.research.att.com (Postfix) with ESMTP id 7DCFBE0237; Wed, 13 Nov 2013 09:40:25 -0500 (EST) Received: from concierge.research.att.com (135.207.255.39) by njfpsrvexg2.research.att.com (135.207.160.21) with Microsoft SMTP Server (TLS) id 8.3.327.1; Wed, 13 Nov 2013 09:40:25 -0500 Received: from NJFPSRVEXG8.research.att.com ([fe80:0000:0000:0000:cdea:b3f6:62.250.24.65]) by concierge.research.att.com ([135.207.24.83]) with mapi; Wed, 13 Nov 2013 09:40:24 -0500 From: "MORTON, ALFRED C (AL)" To: Juergen Schoenwaelder , "Bugenhagen, Michael K" Date: Wed, 13 Nov 2013 09:40:23 -0500 Thread-Topic: [lmap] Comments & opinions on Draft - RE: call for consensus - adoption of draft-burbridge-lmap-information-model as WG document Thread-Index: Ac7f3THp1Dm1HQi3RFCLNFDHSS+x4AAnnGCg Message-ID: <2845723087023D4CB5114223779FA9C8AC0F90BF@njfpsrvexg8.research.att.com> References: <9904FB1B0159DA42B0B887B7FA8119CA1292EAA3@AZ-FFEXMB04.global.avaya.com> <20131111203930.GA770@elstar.local> <20131111214712.GA977@elstar.local> <20131112191106.GA7537@elstar.local> In-Reply-To: <20131112191106.GA7537@elstar.local> Accept-Language: en-US Content-Language: en-US X-MS-Has-Attach: X-MS-TNEF-Correlator: acceptlanguage: en-US Content-Type: text/plain; charset="us-ascii" Content-Transfer-Encoding: quoted-printable MIME-Version: 1.0 X-RSA-Inspected: yes X-RSA-Classifications: public X-Spam: [F=0.2000000000; CM=0.500; S=0.200(2010122901)] X-MAIL-FROM: X-SOURCE-IP: [144.160.229.24] X-AnalysisOut: [v=2.0 cv=GKm/5JxK c=1 sm=0 a=dhB6nF3YHL5t/Ixux6cINA==:17 a] X-AnalysisOut: [=U_gNwDc4ozEA:10 a=ofMgfj31e3cA:10 a=BLceEmwcHowA:10 a=kj9] X-AnalysisOut: [zAlcOel0A:10 a=xqWC_Br6kY4A:10 a=zQP7CpKOAAAA:8 a=o3nl4eRq] X-AnalysisOut: [aZMA:10 a=48vgC7mUAAAA:8 a=e9qsufxtAAAA:8 a=j3Z76cjpAAAA:8] X-AnalysisOut: [ a=L2fjZoOA0N2beC7Ec1kA:9 a=CjuIK1q_8ugA:10 a=FvgKqOQ44qUA] X-AnalysisOut: [:10 a=JrSEOxZJtCQA:10 a=zeshHG33Dl4A:10 a=lZB815dzVvQA:10 ] X-AnalysisOut: [a=W1qU_X6G3J8A:10 a=erIDYRxNvJK_i3mw:21 a=4_-zamZ21lkFCCjj] X-AnalysisOut: [:21] X-Mailman-Approved-At: Wed, 13 Nov 2013 06:47:48 -0800 Cc: "Cook, Charles" , "lmap@ietf.org" , 'Trevor Burbridge' , "'Romascanu, Dan \(Dan\)'" , 'KEN KO' , 'marcelo bagnulo braun' , "STARK, BARBARA H" , "'philip.eardley@bt.com'" Subject: Re: [lmap] Comments & opinions on Draft - RE: call for consensus - adoption of draft-burbridge-lmap-information-model as WG document X-BeenThere: lmap@ietf.org X-Mailman-Version: 2.1.12 Precedence: list List-Id: Large Scale Measurement of Access network Performance List-Unsubscribe: , List-Archive: List-Post: List-Help: List-Subscribe: , X-List-Received-Date: Wed, 13 Nov 2013 14:41:11 -0000 > > js wrote: > > - What registry are you talking about? It seems you are not talking > > about the IPPM metrics registry. So what do you have in mind? >=20 > I'm suggesting we re-align the IPPM metrics registry to fix some of= the large problems being faced by the industry to ensure the registry succ= ess. > I think the needs may have changed in the years the registry is bei= ng worked, and feel we should re-assess the functional role the registry pl= ays. > Personally I haven't been able to match a critical need from the IS= P, or Internet user domain with the registry that will make it a wild sucss= es, but I think it's do-able. js wrote: I assume IPPM folks can make sense out of your request.=20 Al wrote: I'll give this one try (one try only). The current registry design(s) collects entries which are =20 specific instances of metrics, with some input factors specified and some left as free variables to the measurement system (like ipaddrs). This registry has all the details that implementers need to build systems that collect equivalent measurements. There is a clear use for this registry in the LMAP models and protocols. It would be possible to create a separate registry for=20 measurement system implementations that include a sub-set of=20 of the registered instances of metrics. The implementations could be virtua= l: wrote: > 2) If we adopt a registry and we know virtualization is on it's way,=20 > then we need to take that into account (a virtual MA). There is no need to combine these two registries, or advantage that I see. The "registry of implementations" just refers to the "metrics registry"=20 when describing what an implementation can do. Virtual or real makes little difference. /Al > -----Original Message----- > From: Juergen Schoenwaelder [mailto:j.schoenwaelder@jacobs-university.de] > Sent: Tuesday, November 12, 2013 2:11 PM > To: Bugenhagen, Michael K > Cc: Cook, Charles; MORTON, ALFRED C (AL); lmap@ietf.org; 'Trevor > Burbridge'; 'Romascanu, Dan (Dan)'; 'KEN KO'; 'marcelo bagnulo braun'; > STARK, BARBARA H; 'philip.eardley@bt.com' > Subject: Re: [lmap] Comments & opinions on Draft - RE: call for consensus > - adoption of draft-burbridge-lmap-information-model as WG document >=20 > On Tue, Nov 12, 2013 at 03:43:25PM +0000, Bugenhagen, Michael K wrote: > > Juergon - comments in line > > > > > > > I am still trying to understand what you are suggesting. > > > > > > - What does "ISP's need a MA that can Stand by itself as an Overlay" > > > mean? > > > - What does "ISP's ... need one that operates more globally as a > > > universal MA" mean? What is a universal MA? What is more globally > > > here? > > > > > Both of these are answered here - > > > > Three large factors on this one - > > - Opex (operational system cost) > > Simply put to contain OPEX costs ISP's who also offer > VoIP, and Video Over IP like having one test solution. (Brix, Spirent, > ....) > > - Test probe deployment Collision space (if the probes don't > know about each other and don't co-exist well) > > In general most ISP's have three Op's groups (Broadband, > VoIP, and Video over IP) - often each one has it's own system and probes. > > This occurred due to the timeline of when broadband was > a product, then VoIP (new NOC), then Video (yet another NOC and test > system). > > - NFV future virtualization > > Virtualization will enable the convergence of items like > probes on the ISP edge equipment where this testing needs to occur. > > > > Summation - A stand alone system to do LMAP is quickly deployable= , > but drives added cost and Observing the convergence of IP Testing system > > functionality to contain Broadband testing, VoIP testing, and > Video over IP testing vs. One system for each ... It's advisable for > LMAP to be constructed as both a standalone testing overlay, and > part of a converged MA to all the functions. > > I think that's not a problem scope wise provided we recognize > it so the IPPM group can make tests modular in nature vs. probe specific. > > >=20 > I still do not understand very well. If your concern is that MAs are > separate devices that I can tell you that this concern is not > justified. An MA is a function, how it is packaged into devices is > left to implementors. >=20 > > > - What registry are you talking about? It seems you are not talking > > > about the IPPM metrics registry. So what do you have in mind? > > > > I'm suggesting we re-align the IPPM metrics registry to fix some > of the large problems being faced by the industry to ensure the registry > success. > > I think the needs may have changed in the years the registry is > being worked, and feel we should re-assess the functional role the > registry plays. > > Personally I haven't been able to match a critical need from the > ISP, or Internet user domain with the registry that will make it a wild > sucsses, but I think it's do-able. >=20 > I assume IPPM folks can make sense out of your request. >=20 > /js >=20 > -- > Juergen Schoenwaelder Jacobs University Bremen gGmbH > Phone: +49 421 200 3587 Campus Ring 1, 28759 Bremen, Germany > Fax: +49 421 200 3103 From philip.eardley@bt.com Wed Nov 13 08:34:13 2013 Return-Path: X-Original-To: lmap@ietfa.amsl.com Delivered-To: lmap@ietfa.amsl.com Received: from localhost (localhost [127.0.0.1]) by ietfa.amsl.com (Postfix) with ESMTP id C37AA21E812A for ; Wed, 13 Nov 2013 08:34:13 -0800 (PST) X-Virus-Scanned: amavisd-new at amsl.com X-Spam-Flag: NO X-Spam-Score: -103.329 X-Spam-Level: X-Spam-Status: No, score=-103.329 tagged_above=-999 required=5 tests=[AWL=0.270, BAYES_00=-2.599, RCVD_IN_DNSWL_LOW=-1, USER_IN_WHITELIST=-100] Received: from mail.ietf.org ([12.22.58.30]) by localhost (ietfa.amsl.com [127.0.0.1]) (amavisd-new, port 10024) with ESMTP id exancrEpFxsi for ; Wed, 13 Nov 2013 08:34:09 -0800 (PST) Received: from smtpe1.intersmtp.com (smtp64.intersmtp.com [62.239.224.237]) by ietfa.amsl.com (Postfix) with ESMTP id A77B121E80F4 for ; Wed, 13 Nov 2013 08:34:04 -0800 (PST) Received: from EVMHT66-UKRD.domain1.systemhost.net (10.36.3.103) by RDW083A008ED64.smtp-e4.hygiene.service (10.187.98.13) with Microsoft SMTP Server (TLS) id 8.3.298.1; Wed, 13 Nov 2013 16:34:01 +0000 Received: from EMV67-UKRD.domain1.systemhost.net ([169.254.2.147]) by EVMHT66-UKRD.domain1.systemhost.net ([10.36.3.103]) with mapi; Wed, 13 Nov 2013 16:34:01 +0000 From: To: , Date: Wed, 13 Nov 2013 16:32:50 +0000 Thread-Topic: [lmap] Measurement Suppression broadcasting and non-acknowledgement Thread-Index: Ac7fxUKUUuyhzE9GR2KQPQVFSBMUIgAyLvUb Message-ID: References: <2D09D61DDFA73D4C884805CC7865E61130399373@GAALPA1MSGUSR9L.ITServices.sbc.com>, <20131112163500.GD3992@elstar.local> In-Reply-To: <20131112163500.GD3992@elstar.local> Accept-Language: en-US, en-GB Content-Language: en-GB X-MS-Has-Attach: X-MS-TNEF-Correlator: acceptlanguage: en-US, en-GB Content-Type: text/plain; charset="us-ascii" Content-Transfer-Encoding: quoted-printable MIME-Version: 1.0 Cc: lmap@ietf.org Subject: Re: [lmap] Measurement Suppression broadcasting and non-acknowledgement X-BeenThere: lmap@ietf.org X-Mailman-Version: 2.1.12 Precedence: list List-Id: Large Scale Measurement of Access network Performance List-Unsubscribe: , List-Archive: List-Post: List-Help: List-Subscribe: , X-List-Received-Date: Wed, 13 Nov 2013 16:34:13 -0000 Barbara Sounds Sensible, will rewrite to give both possibilities Thanks Phil ________________________________________ From: lmap-bounces@ietf.org [lmap-bounces@ietf.org] On Behalf Of Juergen Sc= hoenwaelder [j.schoenwaelder@jacobs-university.de] Sent: 12 November 2013 16:35 To: STARK, BARBARA H Cc: lmap@ietf.org Subject: Re: [lmap] Measurement Suppression broadcasting and non-acknowledg= ement Barbara, I share your concerns. /js On Tue, Nov 12, 2013 at 04:17:01PM +0000, STARK, BARBARA H wrote: > I have some concerns with the way "Measurement Suppression" is architecte= d in framework currently. > > framework says "the "suppress" message is not acknowledged, since it is l= ikely to be broadcast to several /many MAs at a time when the measurement s= ystem wants to eliminate inessential traffic." > > 1. I'm not sure if this is intended to literally mean "broadcast" (which = is a very specific concept in IP that uses broadcast destination IP address= es like 255.255.255.255 or more specific subnet addresses to reach all host= s on a network with a single sent message), or "multicast" (similar to broa= dcast but sent to a multicast address), or just massive quantities of unica= st messages sent "simultaneously". > Where MAs are internal to an enterprise or campus network (managed by Con= trollers also internal to the network), I could see where literal IP broadc= ast messages (or even multicast messages) might be a good technique to use.= For MAs managed by Controllers that are out on the Internet somewhere (whi= ch I'd hope we all agree are definitely in scope), I don't think it's pract= ical. Therefore I disagree that it's likely. > > 2. I believe it would be a bad idea for MAs managed by Controllers out on= the Internet to react to a suppression message that is unauthenticated and= unencrypted. Therefore, there exists a need for it to be possible for the = Controller and each MA to establish authenticated and encrypted (e.g., TLS)= sessions prior to the Controller sending the suppression command. > > 3. The overhead of having the MA send an Ack message after setting up the= authenticated and encrypted session isn't so big that it should be prohibi= ted, if the Controller operator wants such an Ack to occur. > > Therefore, I disagree that the framework should mandate non-acknowledgeme= nt of suppression messages. I have no problem with unacknowledged suppressi= on messages being allowed -- this can be useful in enterprise/campus enviro= nments. But I also believe there is value in supporting acknowledged suppr= ession messages, and that this option should also be allowed. > _______________________________________________ > lmap mailing list > lmap@ietf.org > https://www.ietf.org/mailman/listinfo/lmap -- Juergen Schoenwaelder Jacobs University Bremen gGmbH Phone: +49 421 200 3587 Campus Ring 1, 28759 Bremen, Germany Fax: +49 421 200 3103 _______________________________________________ lmap mailing list lmap@ietf.org https://www.ietf.org/mailman/listinfo/lmap From Michael.K.Bugenhagen@centurylink.com Wed Nov 13 09:21:38 2013 Return-Path: X-Original-To: lmap@ietfa.amsl.com Delivered-To: lmap@ietfa.amsl.com Received: from localhost (localhost [127.0.0.1]) by ietfa.amsl.com (Postfix) with ESMTP id 5781D11E8152 for ; Wed, 13 Nov 2013 09:21:38 -0800 (PST) X-Virus-Scanned: amavisd-new at amsl.com X-Spam-Flag: NO X-Spam-Score: -2.599 X-Spam-Level: X-Spam-Status: No, score=-2.599 tagged_above=-999 required=5 tests=[AWL=0.000, BAYES_00=-2.599] Received: from mail.ietf.org ([12.22.58.30]) by localhost (ietfa.amsl.com [127.0.0.1]) (amavisd-new, port 10024) with ESMTP id Zy2mwBO+Tczr for ; Wed, 13 Nov 2013 09:21:32 -0800 (PST) Received: from suomp64i.qwest.com (suomp64i.qwest.com [155.70.16.237]) by ietfa.amsl.com (Postfix) with ESMTP id AE00111E81A6 for ; Wed, 13 Nov 2013 09:21:32 -0800 (PST) Received: from lxomavmpc030.qintra.com (lxomavmpc030.qintra.com [151.117.207.30]) by suomp64i.qwest.com (8.14.4/8.14.4) with ESMTP id rADHLMtD025143 (version=TLSv1/SSLv3 cipher=DHE-RSA-AES256-SHA bits=256 verify=NO); Wed, 13 Nov 2013 11:21:22 -0600 (CST) Received: from lxomavmpc030.qintra.com (unknown [127.0.0.1]) by IMSA (Postfix) with ESMTP id 652731E0059; Wed, 13 Nov 2013 11:21:17 -0600 (CST) Received: from sudnp797.qintra.com (unknown [10.6.10.61]) by lxomavmpc030.qintra.com (Postfix) with ESMTP id 127C71E005B; Wed, 13 Nov 2013 11:21:17 -0600 (CST) Received: from sudnp797.qintra.com (localhost [127.0.0.1]) by sudnp797.qintra.com (8.14.4/8.14.4) with ESMTP id rADHLGWr009057; Wed, 13 Nov 2013 10:21:16 -0700 (MST) Received: from vodcwhubex502.ctl.intranet (vodcwhubex502.ctl.intranet [151.117.206.28]) by sudnp797.qintra.com (8.14.4/8.14.4) with ESMTP id rADHLGSo009051 (version=TLSv1/SSLv3 cipher=AES128-SHA bits=128 verify=FAIL); Wed, 13 Nov 2013 10:21:16 -0700 (MST) Received: from PODCWMBXEX505.ctl.intranet ([fe80::f87e:fe44:ad72:b610]) by vodcwhubex502.ctl.intranet ([2002:9775:ce1c::9775:ce1c]) with mapi id 14.02.0318.001; Wed, 13 Nov 2013 11:21:15 -0600 From: "Bugenhagen, Michael K" To: "'Romascanu, Dan (Dan)'" , "'lmap@ietf.org'" , "'Trevor Burbridge'" , "'philip.eardley@bt.com'" , "'marcelo bagnulo braun'" Thread-Topic: Comments & opinions on Draft - RE: [lmap] call for consensus - adoption of draft-burbridge-lmap-information-model as WG document Thread-Index: Ac7bVGevh+4pI7FlSz+236qyIybWMQDqGOVAACVfjoAAQE9LYA== Date: Wed, 13 Nov 2013 17:21:14 +0000 Message-ID: References: <9904FB1B0159DA42B0B887B7FA8119CA1292EAA3@AZ-FFEXMB04.global.avaya.com> <9904FB1B0159DA42B0B887B7FA8119CA1293A32C@AZ-FFEXMB04.global.avaya.com> In-Reply-To: <9904FB1B0159DA42B0B887B7FA8119CA1293A32C@AZ-FFEXMB04.global.avaya.com> Accept-Language: en-US Content-Language: en-US X-MS-Has-Attach: X-MS-TNEF-Correlator: x-originating-ip: [151.117.206.7] Content-Type: text/plain; charset="us-ascii" Content-Transfer-Encoding: quoted-printable MIME-Version: 1.0 X-CFilter-Loop: Reflected Cc: "'Stark, Barbara'" , "Cook, Charles" , "'MORTON, ALFRED C \(AL\)'" , 'KEN KO' Subject: Re: [lmap] Comments & opinions on Draft - RE: call for consensus - adoption of draft-burbridge-lmap-information-model as WG document X-BeenThere: lmap@ietf.org X-Mailman-Version: 2.1.12 Precedence: list List-Id: Large Scale Measurement of Access network Performance List-Unsubscribe: , List-Archive: List-Post: List-Help: List-Subscribe: , X-List-Received-Date: Wed, 13 Nov 2013 17:21:38 -0000 Dan, I support the document as a draft, but also acknowledge that this docume= nt should be the collector of significant set of attributes that have yet t= o be defined, so the need for the document is paramount, the contents withi= n it today are largely subject to change. Either way we need an information model so I support it. Sorry I took me an extra day to reply -They upgraded my laptop so life is n= ow good again..=20 :) Too cut some of the chatter I'll follow up directly with the authors of = the draft on some items that I think should be reconsidered. Thanks, Mike -----Original Message----- From: Romascanu, Dan (Dan) [mailto:dromasca@avaya.com]=20 Sent: Tuesday, November 12, 2013 4:39 AM To: Bugenhagen, Michael K; lmap@ietf.org; 'Trevor Burbridge'; 'philip.eardl= ey@bt.com'; 'marcelo bagnulo braun' Cc: 'MORTON, ALFRED C (AL)'; Cook, Charles; 'KEN KO'; 'Stark, Barbara' Subject: RE: Comments & opinions on Draft - RE: [lmap] call for consensus -= adoption of draft-burbridge-lmap-information-model as WG document Hi Michael, Thank you for your comments. As I can see the authors started to address th= em.=20 My question (wearing the WG chair hat :-) ) - are you OK with the adoption = of draft-burbridge-lmap-information-model as WG document? Saying 'no' means= that you consider that the document needs quite serious changes before bec= oming a WG document, maybe that you have or you are aware about one ore mor= e alternate proposals to be submitted. Saying 'yes' means the document is o= n the right direction, with changes and improvements that can be made in th= e following iterations.=20 Thanks and Regards, Dan > -----Original Message----- > From: Bugenhagen, Michael K > [mailto:Michael.K.Bugenhagen@centurylink.com] > Sent: Monday, November 11, 2013 7:05 PM > To: Romascanu, Dan (Dan); lmap@ietf.org; 'Trevor Burbridge';=20 > 'philip.eardley@bt.com'; 'marcelo bagnulo braun' > Cc: 'MORTON, ALFRED C (AL)'; Cook, Charles; 'KEN KO'; 'Stark, Barbara' > Subject: Comments & opinions on Draft - RE: [lmap] call for consensus=20 > - adoption of draft-burbridge-lmap-information-model as WG document >=20 > Dan, >=20 > Comment 1. Is written from an ISP Operational team perspective (Scope=20 > says ISP use case so... ) Comment 2. Is future proofing the Registry=20 > concept but it also is in the draft a bit.. >=20 >=20 > 1) ISP use case requirements - (The yet another test system overlay > issue) >=20 > ISP's have multiple IP based "testing systems" / "Measurement=20 > Agents" deployed today, many of which don't co-exist well. > So one ongoing fairly solid ISP requirement that the groups should=20 > strongly consider is the ability of a MA to be open flexible framework=20 > MA that can run both Op's ADHOC test, and performance testing without=20 > two MA's. Example (a MA / probe that can run VoIP, VIDEO, LMAP, DNS=20 > testing, ..... vs. multiple probes doing one each.) >=20 > ** Current draft that means adding the ability to >=20 > ADHOC / Multi-user op's needs for considerations > a) MA capability to both Push and Pull test results > b) MA Radius / Authentication to allow item a) > c) Multiple test capability (OAM, and throughput at the same time by=20 > different Op's teams) > d) Resource availability checking (when you can run more than one test=20 > at a time) > e) MA to MA test capability negotiation (if you are working with a=20 > customers or 3rd party probe) >=20 >=20 >=20 > 2) Future proof the Registry / MA probe concept into Virtual=20 > Applications - (start thinking Virtualization Service Catalog) >=20 > As NFV (network functional virtualization) will likely become=20 > available by the time some of this is being implemented we should=20 > consider adding a "hypervisor" type to the Registry. The thought here=20 > is that Virtual applications come in a default deployable set, *.OVF > files for VMware, ... for KVM, and so on. ** ONE KEY to service > catalogs is a trusted source for the code, one should NEVER (NEVER - > EVER) deploy a virtual appliance from a public source on their=20 > network... >=20 > I think the registry could be a good solution to this issue - provided=20 > we place a high level or safety checks on anything getting loaded into=20 > the registry itself. >=20 >=20 >=20 >=20 >=20 >=20 >=20 >=20 >=20 >=20 >=20 >=20 >=20 >=20 >=20 >=20 >=20 >=20 >=20 >=20 >=20 >=20 >=20 >=20 >=20 >=20 >=20 >=20 >=20 >=20 >=20 >=20 >=20 > -----Original Message----- > From: lmap-bounces@ietf.org [mailto:lmap-bounces@ietf.org] On Behalf=20 > Of Romascanu, Dan (Dan) > Sent: Wednesday, November 06, 2013 6:58 PM > To: lmap@ietf.org > Subject: [lmap] call for consensus - adoption of draft-burbridge-lmap-=20 > information-model as WG document >=20 >=20 > Today in the LMAP meeting we ran a hum on the question whether we=20 > should adopt draft-burbridge-lmap-information-model. The result was=20 > read as showing consensus in favor of the adoption, although there was=20 > at least one opposed opinion expressed by Benoit (as contributor). >=20 > We would like to extend this call to the participants who could not=20 > attend the meeting. Please send your opinions on this question to the=20 > WG mail list before Friday 11/15. >=20 > Thanks and Regards, >=20 > Dan >=20 >=20 >=20 > _______________________________________________ > lmap mailing list > lmap@ietf.org > https://www.ietf.org/mailman/listinfo/lmap From j.schoenwaelder@jacobs-university.de Thu Nov 14 01:26:50 2013 Return-Path: X-Original-To: lmap@ietfa.amsl.com Delivered-To: lmap@ietfa.amsl.com Received: from localhost (localhost [127.0.0.1]) by ietfa.amsl.com (Postfix) with ESMTP id BEC6921F9EF2 for ; Thu, 14 Nov 2013 01:26:49 -0800 (PST) X-Virus-Scanned: amavisd-new at amsl.com X-Spam-Flag: NO X-Spam-Score: -103.206 X-Spam-Level: X-Spam-Status: No, score=-103.206 tagged_above=-999 required=5 tests=[AWL=0.043, BAYES_00=-2.599, HELO_EQ_DE=0.35, RCVD_IN_DNSWL_LOW=-1, USER_IN_WHITELIST=-100] Received: from mail.ietf.org ([12.22.58.30]) by localhost (ietfa.amsl.com [127.0.0.1]) (amavisd-new, port 10024) with ESMTP id jKJHF8kl5rWa for ; Thu, 14 Nov 2013 01:26:43 -0800 (PST) Received: from hermes.jacobs-university.de (hermes.jacobs-university.de [212.201.44.23]) by ietfa.amsl.com (Postfix) with ESMTP id 8C7A221F9DAA for ; Thu, 14 Nov 2013 01:26:33 -0800 (PST) Received: from localhost (demetrius4.jacobs-university.de [212.201.44.49]) by hermes.jacobs-university.de (Postfix) with ESMTP id E10FB2007E; Thu, 14 Nov 2013 10:26:32 +0100 (CET) X-Virus-Scanned: amavisd-new at jacobs-university.de Received: from hermes.jacobs-university.de ([212.201.44.23]) by localhost (demetrius4.jacobs-university.de [212.201.44.32]) (amavisd-new, port 10024) with ESMTP id cYZnkbVvhe04; Thu, 14 Nov 2013 10:26:32 +0100 (CET) Received: from elstar.local (elstar.jacobs.jacobs-university.de [10.50.231.133]) by hermes.jacobs-university.de (Postfix) with ESMTP id 76DEB20081; Thu, 14 Nov 2013 10:26:32 +0100 (CET) Received: by elstar.local (Postfix, from userid 501) id 92A742949A7F; Thu, 14 Nov 2013 10:26:25 +0100 (CET) Date: Thu, 14 Nov 2013 10:26:25 +0100 From: Juergen Schoenwaelder To: philip.eardley@bt.com Message-ID: <20131114092625.GA15158@elstar.local> Mail-Followup-To: philip.eardley@bt.com, lmap@ietf.org References: <2D09D61DDFA73D4C884805CC7865E61130399373@GAALPA1MSGUSR9L.ITServices.sbc.com> <20131112163500.GD3992@elstar.local> MIME-Version: 1.0 Content-Type: text/plain; charset=us-ascii Content-Disposition: inline In-Reply-To: User-Agent: Mutt/1.5.21 (2010-09-15) Cc: lmap@ietf.org Subject: Re: [lmap] Measurement Suppression broadcasting and non-acknowledgement X-BeenThere: lmap@ietf.org X-Mailman-Version: 2.1.12 Precedence: list Reply-To: Juergen Schoenwaelder List-Id: Large Scale Measurement of Access network Performance List-Unsubscribe: , List-Archive: List-Post: List-Help: List-Subscribe: , X-List-Received-Date: Thu, 14 Nov 2013 09:26:50 -0000 Phil, so far I assumed we communicate over TLS/TCP and with that a multicast make no sense at all. If you want to include the option to include multicasts, then you have to answer a number of non-trivial questions: - What is the transport? Since we talk wide area, it needs to be congestion aware. - How is the security provided over the transport? How are group keys managed? - How to make multicasts reliable? How to deal with MA that are temporarily down? - Why support multicasts for suppressions but not other changes that might affect many MAs? There is probably more. My point here (and I think this is also Barbara's point) is that doing multicasts for the sake of saving a bunch of ACKs adds significant challenges and so far I have not heard about a system that does this 'optimization' today. I believe this multicast idea shoud be _removed_ and not be made optional - at least for LMAP 1.0. /js On Wed, Nov 13, 2013 at 04:32:50PM +0000, philip.eardley@bt.com wrote: > Barbara > Sounds Sensible, will rewrite to give both possibilities > Thanks > Phil > ________________________________________ > From: lmap-bounces@ietf.org [lmap-bounces@ietf.org] On Behalf Of Juergen Schoenwaelder [j.schoenwaelder@jacobs-university.de] > Sent: 12 November 2013 16:35 > To: STARK, BARBARA H > Cc: lmap@ietf.org > Subject: Re: [lmap] Measurement Suppression broadcasting and non-acknowledgement > > Barbara, > > I share your concerns. > > /js > > On Tue, Nov 12, 2013 at 04:17:01PM +0000, STARK, BARBARA H wrote: > > I have some concerns with the way "Measurement Suppression" is architected in framework currently. > > > > framework says "the "suppress" message is not acknowledged, since it is likely to be broadcast to several /many MAs at a time when the measurement system wants to eliminate inessential traffic." > > > > 1. I'm not sure if this is intended to literally mean "broadcast" (which is a very specific concept in IP that uses broadcast destination IP addresses like 255.255.255.255 or more specific subnet addresses to reach all hosts on a network with a single sent message), or "multicast" (similar to broadcast but sent to a multicast address), or just massive quantities of unicast messages sent "simultaneously". > > Where MAs are internal to an enterprise or campus network (managed by Controllers also internal to the network), I could see where literal IP broadcast messages (or even multicast messages) might be a good technique to use. For MAs managed by Controllers that are out on the Internet somewhere (which I'd hope we all agree are definitely in scope), I don't think it's practical. Therefore I disagree that it's likely. > > > > 2. I believe it would be a bad idea for MAs managed by Controllers out on the Internet to react to a suppression message that is unauthenticated and unencrypted. Therefore, there exists a need for it to be possible for the Controller and each MA to establish authenticated and encrypted (e.g., TLS) sessions prior to the Controller sending the suppression command. > > > > 3. The overhead of having the MA send an Ack message after setting up the authenticated and encrypted session isn't so big that it should be prohibited, if the Controller operator wants such an Ack to occur. > > > > Therefore, I disagree that the framework should mandate non-acknowledgement of suppression messages. I have no problem with unacknowledged suppression messages being allowed -- this can be useful in enterprise/campus environments. But I also believe there is value in supporting acknowledged suppression messages, and that this option should also be allowed. > > _______________________________________________ > > lmap mailing list > > lmap@ietf.org > > https://www.ietf.org/mailman/listinfo/lmap > > -- > Juergen Schoenwaelder Jacobs University Bremen gGmbH > Phone: +49 421 200 3587 Campus Ring 1, 28759 Bremen, Germany > Fax: +49 421 200 3103 > _______________________________________________ > lmap mailing list > lmap@ietf.org > https://www.ietf.org/mailman/listinfo/lmap -- Juergen Schoenwaelder Jacobs University Bremen gGmbH Phone: +49 421 200 3587 Campus Ring 1, 28759 Bremen, Germany Fax: +49 421 200 3103 From Michael.K.Bugenhagen@centurylink.com Thu Nov 14 04:31:47 2013 Return-Path: X-Original-To: lmap@ietfa.amsl.com Delivered-To: lmap@ietfa.amsl.com Received: from localhost (localhost [127.0.0.1]) by ietfa.amsl.com (Postfix) with ESMTP id CCB1311E8174 for ; Thu, 14 Nov 2013 04:31:47 -0800 (PST) X-Virus-Scanned: amavisd-new at amsl.com X-Spam-Flag: NO X-Spam-Score: -2.599 X-Spam-Level: X-Spam-Status: No, score=-2.599 tagged_above=-999 required=5 tests=[AWL=0.000, BAYES_00=-2.599] Received: from mail.ietf.org ([12.22.58.30]) by localhost (ietfa.amsl.com [127.0.0.1]) (amavisd-new, port 10024) with ESMTP id XpAsYitiNoHG for ; Thu, 14 Nov 2013 04:31:42 -0800 (PST) Received: from sudnp799.qwest.com (sudnp799.qwest.com [155.70.32.99]) by ietfa.amsl.com (Postfix) with ESMTP id D7A5011E813F for ; Thu, 14 Nov 2013 04:31:41 -0800 (PST) Received: from lxdenvmpc030.qintra.com (lxdenvmpc030.qintra.com [10.1.51.30]) by sudnp799.qwest.com (8.14.4/8.14.4) with ESMTP id rAECVUgw022341 (version=TLSv1/SSLv3 cipher=DHE-RSA-AES256-SHA bits=256 verify=NO); Thu, 14 Nov 2013 05:31:30 -0700 (MST) Received: from lxdenvmpc030.qintra.com (unknown [127.0.0.1]) by IMSA (Postfix) with ESMTP id 282251E0061; Thu, 14 Nov 2013 05:31:25 -0700 (MST) Received: from sudnp796.qintra.com (unknown [151.119.91.93]) by lxdenvmpc030.qintra.com (Postfix) with ESMTP id DD27F1E0049; Thu, 14 Nov 2013 05:31:24 -0700 (MST) Received: from sudnp796.qintra.com (localhost [127.0.0.1]) by sudnp796.qintra.com (8.14.4/8.14.4) with ESMTP id rAECVOx4006026; Thu, 14 Nov 2013 05:31:24 -0700 (MST) Received: from vodcwhubex501.ctl.intranet (vodcwhubex501.ctl.intranet [151.117.206.27]) by sudnp796.qintra.com (8.14.4/8.14.4) with ESMTP id rAECVOtj006011 (version=TLSv1/SSLv3 cipher=AES128-SHA bits=128 verify=FAIL); Thu, 14 Nov 2013 05:31:24 -0700 (MST) Received: from PODCWMBXEX505.ctl.intranet ([fe80::f87e:fe44:ad72:b610]) by vodcwhubex501.ctl.intranet ([151.117.206.27]) with mapi id 14.02.0318.001; Thu, 14 Nov 2013 06:31:23 -0600 From: "Bugenhagen, Michael K" To: "'Juergen Schoenwaelder'" Thread-Topic: [lmap] Comments & opinions on Draft - RE: call for consensus - adoption of draft-burbridge-lmap-information-model as WG document Thread-Index: AQHO3x4nwl1kPZbVy0C+w4vn4eORgJogftNggAB2rQCAA7W80A== Date: Thu, 14 Nov 2013 12:31:22 +0000 Message-ID: References: <9904FB1B0159DA42B0B887B7FA8119CA1292EAA3@AZ-FFEXMB04.global.avaya.com> <20131111203930.GA770@elstar.local> <20131111214712.GA977@elstar.local> In-Reply-To: <20131111214712.GA977@elstar.local> Accept-Language: en-US Content-Language: en-US X-MS-Has-Attach: X-MS-TNEF-Correlator: x-originating-ip: [151.117.206.7] Content-Type: text/plain; charset="us-ascii" Content-Transfer-Encoding: quoted-printable MIME-Version: 1.0 X-CFilter-Loop: Reflected X-Mailman-Approved-At: Thu, 14 Nov 2013 04:32:47 -0800 Cc: "Cook, Charles" , "'MORTON, ALFRED C \(AL\)'" , "'lmap@ietf.org'" , 'Trevor Burbridge' , "'Romascanu, Dan \(Dan\)'" , 'KEN KO' , 'marcelo bagnulo braun' , "'Stark, Barbara'" , "'philip.eardley@bt.com'" Subject: Re: [lmap] Comments & opinions on Draft - RE: call for consensus - adoption of draft-burbridge-lmap-information-model as WG document X-BeenThere: lmap@ietf.org X-Mailman-Version: 2.1.12 Precedence: list List-Id: Large Scale Measurement of Access network Performance List-Unsubscribe: , List-Archive: List-Post: List-Help: List-Subscribe: , X-List-Received-Date: Thu, 14 Nov 2013 12:31:47 -0000 Juergen, I'm trying to tell you that the Operations groups in the industry are p= utting probes on the Modems to troubleshoot all the services. If the new LMAP probe can't co-exist with an existing probe that is deploye= d everywhere that would be highly problematic for some ISP's. =20 My 15+ years of Operations troubleshooting experience is often "asked" fo= r in standards discussions -not always appreciated though.. :) Hope that clears up why I'm recommending we make sure isn't a complete test= ing overlay that can't exist with other existing overlays. In other words - If an ISP has a testing / MA / Probe control plane that= 's can't co-exist with this new one... one will lose out. Thanks, Mike -----Original Message----- From: Juergen Schoenwaelder [mailto:j.schoenwaelder@jacobs-university.de]=20 Sent: Monday, November 11, 2013 3:47 PM To: Bugenhagen, Michael K Cc: 'Romascanu, Dan (Dan)'; lmap@ietf.org; 'Trevor Burbridge'; 'philip.eard= ley@bt.com'; 'marcelo bagnulo braun'; 'Stark, Barbara'; Cook, Charles; 'MOR= TON, ALFRED C (AL)'; 'KEN KO' Subject: Re: [lmap] Comments & opinions on Draft - RE: call for consensus -= adoption of draft-burbridge-lmap-information-model as WG document On Mon, Nov 11, 2013 at 08:46:35PM +0000, Bugenhagen, Michael K wrote: > Juergen, >=20 > If you look at the bigger picture my comments are saying we should con= sider those 2 items as requirements of the use cases. > Or at least consider them as such... before running too far into any mode= l that may break if critical requirements are missed. >=20 > The 2 I am suggesting here - >=20 > 1) ISP's need a MA that can Stand by itself as an Overlay, but more impor= tantly they may need one that operates more globally as a universal MA. > 2) If we adopt a registry and we know virtualization is on it's way, then= we need to take that into account (a virtual MA). >=20 I am still trying to understand what you are suggesting.=20 - What does "ISP's need a MA that can Stand by itself as an Overlay" mean? - What does "ISP's ... need one that operates more globally as a universal MA" mean? What is a universal MA? What is more globally here? - What registry are you talking about? It seems you are not talking about the IPPM metrics registry. So what do you have in mind? Reading again your other email, if you are talking about an MA under contro= l of multiple controllers, then I am afraid LMAP 1.0 won't address this (se= e the WG's charter). But MAs can of course implement tests for multiple met= rics. /js --=20 Juergen Schoenwaelder Jacobs University Bremen gGmbH Phone: +49 421 200 3587 Campus Ring 1, 28759 Bremen, Germany Fax: +49 421 200 3103 From j.schoenwaelder@jacobs-university.de Thu Nov 14 04:35:35 2013 Return-Path: X-Original-To: lmap@ietfa.amsl.com Delivered-To: lmap@ietfa.amsl.com Received: from localhost (localhost [127.0.0.1]) by ietfa.amsl.com (Postfix) with ESMTP id DF3F411E813F for ; Thu, 14 Nov 2013 04:35:33 -0800 (PST) X-Virus-Scanned: amavisd-new at amsl.com X-Spam-Flag: NO X-Spam-Score: -103.207 X-Spam-Level: X-Spam-Status: No, score=-103.207 tagged_above=-999 required=5 tests=[AWL=0.042, BAYES_00=-2.599, HELO_EQ_DE=0.35, RCVD_IN_DNSWL_LOW=-1, USER_IN_WHITELIST=-100] Received: from mail.ietf.org ([12.22.58.30]) by localhost (ietfa.amsl.com [127.0.0.1]) (amavisd-new, port 10024) with ESMTP id Nm1mga8-Gpri for ; Thu, 14 Nov 2013 04:35:29 -0800 (PST) Received: from hermes.jacobs-university.de (hermes.jacobs-university.de [212.201.44.23]) by ietfa.amsl.com (Postfix) with ESMTP id CB4A411E8174 for ; Thu, 14 Nov 2013 04:35:28 -0800 (PST) Received: from localhost (demetrius3.jacobs-university.de [212.201.44.48]) by hermes.jacobs-university.de (Postfix) with ESMTP id 242E82007E; Thu, 14 Nov 2013 13:35:28 +0100 (CET) X-Virus-Scanned: amavisd-new at jacobs-university.de Received: from hermes.jacobs-university.de ([212.201.44.23]) by localhost (demetrius3.jacobs-university.de [212.201.44.32]) (amavisd-new, port 10024) with ESMTP id it2EXO7gk9Ae; Thu, 14 Nov 2013 13:35:28 +0100 (CET) Received: from elstar.local (elstar.jacobs.jacobs-university.de [10.50.231.133]) by hermes.jacobs-university.de (Postfix) with ESMTP id 88FF220055; Thu, 14 Nov 2013 13:35:25 +0100 (CET) Received: by elstar.local (Postfix, from userid 501) id E051F294A555; Thu, 14 Nov 2013 13:35:20 +0100 (CET) Date: Thu, 14 Nov 2013 13:35:20 +0100 From: Juergen Schoenwaelder To: trevor.burbridge@bt.com Message-ID: <20131114123520.GB15852@elstar.local> Mail-Followup-To: trevor.burbridge@bt.com, steve@idrathernotsay.com, lmap@ietf.org References: <20131111182839.GA21454@idrathernotsay.com> <20131111204218.GB770@elstar.local> <20131112185611.GA33370@idrathernotsay.com> <20131112192555.GB7537@elstar.local> MIME-Version: 1.0 Content-Type: text/plain; charset=us-ascii Content-Disposition: inline In-Reply-To: User-Agent: Mutt/1.5.21 (2010-09-15) Cc: steve@idrathernotsay.com, lmap@ietf.org Subject: Re: [lmap] Splitting measurement tasks into measurement tasks plus reporting tasks? X-BeenThere: lmap@ietf.org X-Mailman-Version: 2.1.12 Precedence: list Reply-To: Juergen Schoenwaelder List-Id: Large Scale Measurement of Access network Performance List-Unsubscribe: , List-Archive: List-Post: List-Help: List-Subscribe: , X-List-Received-Date: Thu, 14 Nov 2013 12:35:36 -0000 Trevor, concerning (b) and (c), I think we should not consider this as a requirement for LMAP 1.0 but perhaps we should consider how such filtering and processing steps could be added in a later version of LMAP. In other words, we may want to think about how we can make the information model extensible such that (b) and (c) could be added nicely in a later phase. /js On Wed, Nov 13, 2013 at 08:53:41AM +0000, trevor.burbridge@bt.com wrote: > (d) we have in the information model. > > (a)-(c) are interesting questions. (a) could always be a configurable parameter. (b) and (c) I've kept away from for reasons of complexity, but they could be useful tools. My initial reaction is that we don't need pre-processing as (i) the amount of data returned is not likely to be huge in most scenarios; (ii) some of this can be done within the test - for example a stream of ping packets can be a single measurement and report avg, min, max etc. > > Trevor. > > >-----Original Message----- > >From: lmap-bounces@ietf.org [mailto:lmap-bounces@ietf.org] On Behalf Of > >Juergen Schoenwaelder > >Sent: 12 November 2013 19:26 > >To: Steve Miller > >Cc: lmap@ietf.org > >Subject: Re: [lmap] Splitting measurement tasks into measurement tasks plus > >reporting tasks? > > > >I think you touch on several issues: > > > >a) Are reports send if nothing is pending to be reported? > > > > I think the answer should be no but this can be discussed. > > > >b) Should we support a notion of filters that can be applied, that is > > only result records passing a filter expression are actually reported. > > > >c) In a previous email, someone suggested to have aggregation > > functions that can be applied generically to test results > > (e.g., calculation of avg, min, max, ...). > > > >d) Should we support a report timing that kind of says "report > > immediately if a result record is available" > > > >These are all fair questions one can ask and it is going to be a > >trade-off between simplicity and functional richness. Perhaps we need > >to look at what deployed systems really do so we do not get over board > >building aggregation and filtering features into the model that can > >also easily be applied on the collector's end. > > > >/js > > > >On Tue, Nov 12, 2013 at 01:56:11PM -0500, Steve Miller wrote: > >> I think that's close but I'm not quite sure it's right. I think part of the gotcha > >here is that the spec tends to express Measurement Tasks as things that have a > >definite start and end time, while I'm thinking of things that are running > >continuously. > >> > >> That is, if we wanted to do a DNS query once a second, forever, the current > >model would express that more as: > >> > >> Measurement Task "DNS Performance": Perform a DNS > >loss/latency/jitter check once a second, for 300 seconds, reporting avg/min/max > >loss, latency and jitter plus a histogram of the latencies. > >> > >> Report Channel "Collector C": Report to collector at > >0,5,10,15,20,25,30,35,40,45,50,55 minutes after the hour. > >> > >> Measurement Schedule: Run the Measurement Task "DNS Performance" > >at 0,5,10,15,20,25,30,35,40,45,50,55 minutes after the hour and report using > >Report Channel "Collector C". > >> > >> but each of those reports has a very definite start and end time -- there's no > >way there to define a report via a rolling window. > >> > >> In the current framework, to get what I'm thinking about, I can *almost* do > >it, but I'd have to do that separately. First, to get the performance numbers every > >five minutes, we'd have what's above, more or less: > >> > >> Measurement Task "DNS Performance": Perform a DNS > >loss/latency/jitter check once a second, for 300 seconds, reporting avg/min/max > >loss, latency and jitter plus a histogram of the latencies. > >> > >> Report Channel "Collector C 300Secs": Report to collector at > >0,5,10,15,20,25,30,35,40,45,50,55 minutes after the hour. > >> > >> Measurement Schedule: Run the Measurement Task "DNS Performance" > >at 0,5,10,15,20,25,30,35,40,45,50,55 minutes after the hour and report using > >Report Channel "Collector C 300Secs". > >> > >> and then to get some sort of report immediately, it seems like about as close as > >we'd get would be: > >> > >> Measurement Task "DNS Badness": Perform a DNS loss/latency/jitter > >check once a second, for 300 seconds, reporting avg/min/max loss, latency and > >jitter immediately if the loss is worse than 1%, the latency is more than 200ms, or > >the jitter is more than 200ms. > >> > >> Report Channel "Collector C 1Sec": Report to collector once a second. > >> > >> Measurement Schedule: Run the Measurement Task "DNS Badness" at > >0,5,10,15,20,25,30,35,40,45,50,55 minutes after the hour and report using > >Report Channel "Collector C 1Sec". > >> > >> which has a couple of issues: > >> > >> - now we're sending twice as much traffic to each destination as we were > >before (which in this example isn't such a big deal but it's not hard to imagine that > >a file-download test or an RTP test wouldn't be so forgiving (-: ) > >> > >> - it's not truly a rolling window, so you either have to clamp the > >accelerated reporting until you have enough baseline to be useful (that is, you > >don't want to report "OMG! 100% loss!" if just the first packet is lost) or you have > >to put up with noisy reports > >> > >> - I think the collector gets a report every second, even if that report is > >empty > >> > >> Maybe a way to look at this would be to have the concepts of: > >> > >> - never-ending measurements, which run continuously > >> > >> and > >> > >> - async report channels, where you could express simple conditions to > >control whether or not a report occurs > >> > >> or maybe async report channels are really some sort of async measurement > >schedule: > >> > >> Run the test named "DNS Badness" continuously; report using the > >channel "Collector C" whenever loss is more than 1% or the latency is more than > >200ms or the jitter is more than 200ms over the last 300 seconds. > >> > >> I could be convinced that I'm thinking too hard about this or that my > >perception of the measurement model as viewing Measurement Tasks as discrete > >things with discrete start times and durations is simply incorrect. > >> > >> Did that make anything more clear? > >> > >> Also, to address some of what you and Trevor were discussing: I don't know > >that I necessarily regard these async reports as logging or status. I think they're > >closer to "normal" reports, they just happen asynchronously. I could be > >convinced to treat this as logging or status, too, if that'd be helpful somehow. > >> > >> -Steve > >> > >> On Mon, Nov 11, 2013 at 09:42:18PM +0100, Juergen Schoenwaelder wrote: > >> > I hear two suggestions here: > >> > > >> > a) Give reporting channels their own independent schedule (right now > >> > they are essentially tied to the scheduled task). > >> > > >> > b) Provide a means for post-processing raw data in order to derive > >> > certain statistics before the result is shipped via a report > >> > channel. > >> > > >> > Is this a fair summary of the request? > >> _______________________________________________ > >> lmap mailing list > >> lmap@ietf.org > >> https://www.ietf.org/mailman/listinfo/lmap > > > >-- > >Juergen Schoenwaelder Jacobs University Bremen gGmbH > >Phone: +49 421 200 3587 Campus Ring 1, 28759 Bremen, Germany > >Fax: +49 421 200 3103 > >_______________________________________________ > >lmap mailing list > >lmap@ietf.org > >https://www.ietf.org/mailman/listinfo/lmap -- Juergen Schoenwaelder Jacobs University Bremen gGmbH Phone: +49 421 200 3587 Campus Ring 1, 28759 Bremen, Germany Fax: +49 421 200 3103 From Michael.K.Bugenhagen@centurylink.com Thu Nov 14 04:40:31 2013 Return-Path: X-Original-To: lmap@ietfa.amsl.com Delivered-To: lmap@ietfa.amsl.com Received: from localhost (localhost [127.0.0.1]) by ietfa.amsl.com (Postfix) with ESMTP id 3075E21E8085 for ; Thu, 14 Nov 2013 04:40:31 -0800 (PST) X-Virus-Scanned: amavisd-new at amsl.com X-Spam-Flag: NO X-Spam-Score: -2.599 X-Spam-Level: X-Spam-Status: No, score=-2.599 tagged_above=-999 required=5 tests=[AWL=0.000, BAYES_00=-2.599] Received: from mail.ietf.org ([12.22.58.30]) by localhost (ietfa.amsl.com [127.0.0.1]) (amavisd-new, port 10024) with ESMTP id JTmovaQdq6KX for ; Thu, 14 Nov 2013 04:40:26 -0800 (PST) Received: from sudnp799.qwest.com (sudnp799.qwest.com [155.70.32.99]) by ietfa.amsl.com (Postfix) with ESMTP id F2A7311E8174 for ; Thu, 14 Nov 2013 04:40:25 -0800 (PST) Received: from lxdenvmpc030.qintra.com (lxdenvmpc030.qintra.com [10.1.51.30]) by sudnp799.qwest.com (8.14.4/8.14.4) with ESMTP id rAECe9qH025394 (version=TLSv1/SSLv3 cipher=DHE-RSA-AES256-SHA bits=256 verify=NO); Thu, 14 Nov 2013 05:40:09 -0700 (MST) Received: from lxdenvmpc030.qintra.com (unknown [127.0.0.1]) by IMSA (Postfix) with ESMTP id 1E7C91E0053; Thu, 14 Nov 2013 05:40:04 -0700 (MST) Received: from sudnp797.qintra.com (unknown [151.119.91.93]) by lxdenvmpc030.qintra.com (Postfix) with ESMTP id 021301E004E; Thu, 14 Nov 2013 05:40:04 -0700 (MST) Received: from sudnp797.qintra.com (localhost [127.0.0.1]) by sudnp797.qintra.com (8.14.4/8.14.4) with ESMTP id rAECe3rP010236; Thu, 14 Nov 2013 05:40:03 -0700 (MST) Received: from vodcwhubex501.ctl.intranet (vodcwhubex501.ctl.intranet [151.117.206.27]) by sudnp797.qintra.com (8.14.4/8.14.4) with ESMTP id rAECe3ba010160 (version=TLSv1/SSLv3 cipher=AES128-SHA bits=128 verify=FAIL); Thu, 14 Nov 2013 05:40:03 -0700 (MST) Received: from PODCWMBXEX505.ctl.intranet ([fe80::f87e:fe44:ad72:b610]) by vodcwhubex501.ctl.intranet ([151.117.206.27]) with mapi id 14.02.0318.001; Thu, 14 Nov 2013 06:40:02 -0600 From: "Bugenhagen, Michael K" To: "'Juergen Schoenwaelder'" , "'trevor.burbridge@bt.com'" Thread-Topic: [lmap] Splitting measurement tasks into measurement tasks plus reporting tasks? Thread-Index: AQHO3wvojra8Di0Ei0Ow1BzayfHDx5og44MAgAF0r4CAAAhOgIAA4bCAgAHQQwD//5vgMA== Date: Thu, 14 Nov 2013 12:40:01 +0000 Message-ID: References: <20131111182839.GA21454@idrathernotsay.com> <20131111204218.GB770@elstar.local> <20131112185611.GA33370@idrathernotsay.com> <20131112192555.GB7537@elstar.local> <20131114123520.GB15852@elstar.local> In-Reply-To: <20131114123520.GB15852@elstar.local> Accept-Language: en-US Content-Language: en-US X-MS-Has-Attach: X-MS-TNEF-Correlator: x-originating-ip: [151.117.206.7] Content-Type: text/plain; charset="us-ascii" Content-Transfer-Encoding: quoted-printable MIME-Version: 1.0 X-CFilter-Loop: Reflected Cc: "'steve@idrathernotsay.com'" , "'lmap@ietf.org'" Subject: Re: [lmap] Splitting measurement tasks into measurement tasks plus reporting tasks? X-BeenThere: lmap@ietf.org X-Mailman-Version: 2.1.12 Precedence: list List-Id: Large Scale Measurement of Access network Performance List-Unsubscribe: , List-Archive: List-Post: List-Help: List-Subscribe: , X-List-Received-Date: Thu, 14 Nov 2013 12:40:31 -0000 >From a functional perspective the Controller interface needs tighter securi= ty controls (Keys), but the collectors do not. To that effect splitting them makes good sense to me. If that's too difficult up front, it just makes implementation more diffi= cult and costly. Just my 2 cents... -----Original Message----- From: lmap-bounces@ietf.org [mailto:lmap-bounces@ietf.org] On Behalf Of Jue= rgen Schoenwaelder Sent: Thursday, November 14, 2013 6:35 AM To: trevor.burbridge@bt.com Cc: steve@idrathernotsay.com; lmap@ietf.org Subject: Re: [lmap] Splitting measurement tasks into measurement tasks plus= reporting tasks? Trevor, concerning (b) and (c), I think we should not consider this as a requiremen= t for LMAP 1.0 but perhaps we should consider how such filtering and proces= sing steps could be added in a later version of LMAP. In other words, we ma= y want to think about how we can make the information model extensible such= that (b) and (c) could be added nicely in a later phase. /js On Wed, Nov 13, 2013 at 08:53:41AM +0000, trevor.burbridge@bt.com wrote: > (d) we have in the information model. >=20 > (a)-(c) are interesting questions. (a) could always be a configurable par= ameter. (b) and (c) I've kept away from for reasons of complexity, but they= could be useful tools. My initial reaction is that we don't need pre-proce= ssing as (i) the amount of data returned is not likely to be huge in most s= cenarios; (ii) some of this can be done within the test - for example a str= eam of ping packets can be a single measurement and report avg, min, max et= c. >=20 > Trevor. >=20 > >-----Original Message----- > >From: lmap-bounces@ietf.org [mailto:lmap-bounces@ietf.org] On Behalf=20 > >Of Juergen Schoenwaelder > >Sent: 12 November 2013 19:26 > >To: Steve Miller > >Cc: lmap@ietf.org > >Subject: Re: [lmap] Splitting measurement tasks into measurement=20 > >tasks plus reporting tasks? > > > >I think you touch on several issues: > > > >a) Are reports send if nothing is pending to be reported? > > > > I think the answer should be no but this can be discussed. > > > >b) Should we support a notion of filters that can be applied, that is > > only result records passing a filter expression are actually reported= . > > > >c) In a previous email, someone suggested to have aggregation > > functions that can be applied generically to test results > > (e.g., calculation of avg, min, max, ...). > > > >d) Should we support a report timing that kind of says "report > > immediately if a result record is available" > > > >These are all fair questions one can ask and it is going to be a=20 > >trade-off between simplicity and functional richness. Perhaps we need=20 > >to look at what deployed systems really do so we do not get over=20 > >board building aggregation and filtering features into the model that=20 > >can also easily be applied on the collector's end. > > > >/js > > > >On Tue, Nov 12, 2013 at 01:56:11PM -0500, Steve Miller wrote: > >> I think that's close but I'm not quite sure it's right. I think=20 > >> part of the gotcha > >here is that the spec tends to express Measurement Tasks as things=20 > >that have a definite start and end time, while I'm thinking of things=20 > >that are running continuously. > >> > >> That is, if we wanted to do a DNS query once a second, forever,=20 > >> the current > >model would express that more as: > >> > >> Measurement Task "DNS Performance": Perform a DNS > >loss/latency/jitter check once a second, for 300 seconds, reporting=20 > >avg/min/max loss, latency and jitter plus a histogram of the latencies. > >> > >> Report Channel "Collector C": Report to collector at > >0,5,10,15,20,25,30,35,40,45,50,55 minutes after the hour. > >> > >> Measurement Schedule: Run the Measurement Task "DNS Performance" > >at 0,5,10,15,20,25,30,35,40,45,50,55 minutes after the hour and=20 > >report using Report Channel "Collector C". > >> > >> but each of those reports has a very definite start and end time --=20 > >> there's no > >way there to define a report via a rolling window. > >> > >> In the current framework, to get what I'm thinking about, I can=20 > >> *almost* do > >it, but I'd have to do that separately. First, to get the=20 > >performance numbers every five minutes, we'd have what's above, more or = less: > >> > >> Measurement Task "DNS Performance": Perform a DNS > >loss/latency/jitter check once a second, for 300 seconds, reporting=20 > >avg/min/max loss, latency and jitter plus a histogram of the latencies. > >> > >> Report Channel "Collector C 300Secs": Report to collector at > >0,5,10,15,20,25,30,35,40,45,50,55 minutes after the hour. > >> > >> Measurement Schedule: Run the Measurement Task "DNS Performance" > >at 0,5,10,15,20,25,30,35,40,45,50,55 minutes after the hour and=20 > >report using Report Channel "Collector C 300Secs". > >> > >> and then to get some sort of report immediately, it seems like=20 > >> about as close as > >we'd get would be: > >> > >> Measurement Task "DNS Badness": Perform a DNS loss/latency/jitter > >check once a second, for 300 seconds, reporting avg/min/max loss,=20 > >latency and jitter immediately if the loss is worse than 1%, the=20 > >latency is more than 200ms, or the jitter is more than 200ms. > >> > >> Report Channel "Collector C 1Sec": Report to collector once a second. > >> > >> Measurement Schedule: Run the Measurement Task "DNS Badness" at > >0,5,10,15,20,25,30,35,40,45,50,55 minutes after the hour and report=20 > >using Report Channel "Collector C 1Sec". > >> > >> which has a couple of issues: > >> > >> - now we're sending twice as much traffic to each destination as=20 > >> we were > >before (which in this example isn't such a big deal but it's not hard=20 > >to imagine that a file-download test or an RTP test wouldn't be so=20 > >forgiving (-: ) > >> > >> - it's not truly a rolling window, so you either have to clamp the > >accelerated reporting until you have enough baseline to be useful=20 > >(that is, you don't want to report "OMG! 100% loss!" if just the=20 > >first packet is lost) or you have to put up with noisy reports > >> > >> - I think the collector gets a report every second, even if that=20 > >> report is > >empty > >> > >> Maybe a way to look at this would be to have the concepts of: > >> > >> - never-ending measurements, which run continuously > >> > >> and > >> > >> - async report channels, where you could express simple conditions=20 > >> to > >control whether or not a report occurs > >> > >> or maybe async report channels are really some sort of async=20 > >> measurement > >schedule: > >> > >> Run the test named "DNS Badness" continuously; report using the > >channel "Collector C" whenever loss is more than 1% or the latency is=20 > >more than 200ms or the jitter is more than 200ms over the last 300 secon= ds. > >> > >> I could be convinced that I'm thinking too hard about this or=20 > >> that my > >perception of the measurement model as viewing Measurement Tasks as=20 > >discrete things with discrete start times and durations is simply incorr= ect. > >> > >> Did that make anything more clear? > >> > >> Also, to address some of what you and Trevor were discussing: I=20 > >> don't know > >that I necessarily regard these async reports as logging or status. =20 > >I think they're closer to "normal" reports, they just happen=20 > >asynchronously. I could be convinced to treat this as logging or status= , too, if that'd be helpful somehow. > >> > >> -Steve > >> > >> On Mon, Nov 11, 2013 at 09:42:18PM +0100, Juergen Schoenwaelder wrote: > >> > I hear two suggestions here: > >> > > >> > a) Give reporting channels their own independent schedule (right now > >> > they are essentially tied to the scheduled task). > >> > > >> > b) Provide a means for post-processing raw data in order to derive > >> > certain statistics before the result is shipped via a report > >> > channel. > >> > > >> > Is this a fair summary of the request? > >> _______________________________________________ > >> lmap mailing list > >> lmap@ietf.org > >> https://www.ietf.org/mailman/listinfo/lmap > > > >-- > >Juergen Schoenwaelder Jacobs University Bremen gGmbH > >Phone: +49 421 200 3587 Campus Ring 1, 28759 Bremen, Germany > >Fax: +49 421 200 3103 > >_______________________________________________ > >lmap mailing list > >lmap@ietf.org > >https://www.ietf.org/mailman/listinfo/lmap --=20 Juergen Schoenwaelder Jacobs University Bremen gGmbH Phone: +49 421 200 3587 Campus Ring 1, 28759 Bremen, Germany Fax: +49 421 200 3103 _______________________________________________ lmap mailing list lmap@ietf.org https://www.ietf.org/mailman/listinfo/lmap From j.schoenwaelder@jacobs-university.de Thu Nov 14 04:42:29 2013 Return-Path: X-Original-To: lmap@ietfa.amsl.com Delivered-To: lmap@ietfa.amsl.com Received: from localhost (localhost [127.0.0.1]) by ietfa.amsl.com (Postfix) with ESMTP id 08D7F11E81A7 for ; Thu, 14 Nov 2013 04:42:29 -0800 (PST) X-Virus-Scanned: amavisd-new at amsl.com X-Spam-Flag: NO X-Spam-Score: -103.207 X-Spam-Level: X-Spam-Status: No, score=-103.207 tagged_above=-999 required=5 tests=[AWL=0.042, BAYES_00=-2.599, HELO_EQ_DE=0.35, RCVD_IN_DNSWL_LOW=-1, USER_IN_WHITELIST=-100] Received: from mail.ietf.org ([12.22.58.30]) by localhost (ietfa.amsl.com [127.0.0.1]) (amavisd-new, port 10024) with ESMTP id mkJSvQViA7+b for ; Thu, 14 Nov 2013 04:42:23 -0800 (PST) Received: from hermes.jacobs-university.de (hermes.jacobs-university.de [212.201.44.23]) by ietfa.amsl.com (Postfix) with ESMTP id BF38621E8085 for ; Thu, 14 Nov 2013 04:42:20 -0800 (PST) Received: from localhost (demetrius1.jacobs-university.de [212.201.44.46]) by hermes.jacobs-university.de (Postfix) with ESMTP id 186362007A; Thu, 14 Nov 2013 13:42:20 +0100 (CET) X-Virus-Scanned: amavisd-new at jacobs-university.de Received: from hermes.jacobs-university.de ([212.201.44.23]) by localhost (demetrius1.jacobs-university.de [212.201.44.32]) (amavisd-new, port 10024) with ESMTP id by7N0BsZ5mik; Thu, 14 Nov 2013 13:42:19 +0100 (CET) Received: from elstar.local (elstar.jacobs.jacobs-university.de [10.50.231.133]) by hermes.jacobs-university.de (Postfix) with ESMTP id EDBC120055; Thu, 14 Nov 2013 13:42:18 +0100 (CET) Received: by elstar.local (Postfix, from userid 501) id 448E7294A5CF; Thu, 14 Nov 2013 13:42:13 +0100 (CET) Date: Thu, 14 Nov 2013 13:42:12 +0100 From: Juergen Schoenwaelder To: "Bugenhagen, Michael K" Message-ID: <20131114124211.GC15852@elstar.local> Mail-Followup-To: "Bugenhagen, Michael K" , "'Romascanu, Dan (Dan)'" , "'lmap@ietf.org'" , 'Trevor Burbridge' , "'philip.eardley@bt.com'" , 'marcelo bagnulo braun' , "'Stark, Barbara'" , "Cook, Charles" , "'MORTON, ALFRED C (AL)'" , 'KEN KO' References: <9904FB1B0159DA42B0B887B7FA8119CA1292EAA3@AZ-FFEXMB04.global.avaya.com> <20131111203930.GA770@elstar.local> <20131111214712.GA977@elstar.local> MIME-Version: 1.0 Content-Type: text/plain; charset=us-ascii Content-Disposition: inline In-Reply-To: User-Agent: Mutt/1.5.21 (2010-09-15) X-Mailman-Approved-At: Thu, 14 Nov 2013 05:14:38 -0800 Cc: "Cook, Charles" , "'MORTON, ALFRED C \(AL\)'" , "'lmap@ietf.org'" , 'Trevor Burbridge' , "'Romascanu, Dan \(Dan\)'" , 'KEN KO' , 'marcelo bagnulo braun' , "'Stark, Barbara'" , "'philip.eardley@bt.com'" Subject: Re: [lmap] Comments & opinions on Draft - RE: call for consensus - adoption of draft-burbridge-lmap-information-model as WG document X-BeenThere: lmap@ietf.org X-Mailman-Version: 2.1.12 Precedence: list Reply-To: Juergen Schoenwaelder List-Id: Large Scale Measurement of Access network Performance List-Unsubscribe: , List-Archive: List-Post: List-Help: List-Subscribe: , X-List-Received-Date: Thu, 14 Nov 2013 12:42:29 -0000 Mike, I appreciate your input I just have trouble to get at the core of it and hence we need several iterations. I now understand that you say "it is essential that LMAP probes can co-exist with existing probes already deployed out there" (trying to get rid of double negations). Correct me if this is wrong. If this is your point, I fully agree. Now the engineering question of course is: what does 'can co-exist' mean? What should LMAP try to avoid because this would make co-existance difficult? What should LMAP consider to make co-existance easier? Can we turn the rather high-level statement into more concrete requirements? Or is there anything specific in the I-Ds that makes you believe LMAP probes would not be able to co-exist? /js On Thu, Nov 14, 2013 at 12:31:22PM +0000, Bugenhagen, Michael K wrote: > Juergen, > > I'm trying to tell you that the Operations groups in the industry are putting probes on the Modems to troubleshoot all the services. > If the new LMAP probe can't co-exist with an existing probe that is deployed everywhere that would be highly problematic for some ISP's. > > My 15+ years of Operations troubleshooting experience is often "asked" for in standards discussions -not always appreciated though.. > > :) > > Hope that clears up why I'm recommending we make sure isn't a complete testing overlay that can't exist with other existing overlays. > In other words - If an ISP has a testing / MA / Probe control plane that's can't co-exist with this new one... one will lose out. > > Thanks, > Mike > > > > -----Original Message----- > From: Juergen Schoenwaelder [mailto:j.schoenwaelder@jacobs-university.de] > Sent: Monday, November 11, 2013 3:47 PM > To: Bugenhagen, Michael K > Cc: 'Romascanu, Dan (Dan)'; lmap@ietf.org; 'Trevor Burbridge'; 'philip.eardley@bt.com'; 'marcelo bagnulo braun'; 'Stark, Barbara'; Cook, Charles; 'MORTON, ALFRED C (AL)'; 'KEN KO' > Subject: Re: [lmap] Comments & opinions on Draft - RE: call for consensus - adoption of draft-burbridge-lmap-information-model as WG document > > On Mon, Nov 11, 2013 at 08:46:35PM +0000, Bugenhagen, Michael K wrote: > > Juergen, > > > > If you look at the bigger picture my comments are saying we should consider those 2 items as requirements of the use cases. > > Or at least consider them as such... before running too far into any model that may break if critical requirements are missed. > > > > The 2 I am suggesting here - > > > > 1) ISP's need a MA that can Stand by itself as an Overlay, but more importantly they may need one that operates more globally as a universal MA. > > 2) If we adopt a registry and we know virtualization is on it's way, then we need to take that into account (a virtual MA). > > > > I am still trying to understand what you are suggesting. > > - What does "ISP's need a MA that can Stand by itself as an Overlay" > mean? > > - What does "ISP's ... need one that operates more globally as a > universal MA" mean? What is a universal MA? What is more globally > here? > > - What registry are you talking about? It seems you are not talking > about the IPPM metrics registry. So what do you have in mind? > > Reading again your other email, if you are talking about an MA under control of multiple controllers, then I am afraid LMAP 1.0 won't address this (see the WG's charter). But MAs can of course implement tests for multiple metrics. > > /js > > -- > Juergen Schoenwaelder Jacobs University Bremen gGmbH > Phone: +49 421 200 3587 Campus Ring 1, 28759 Bremen, Germany > Fax: +49 421 200 3103 -- Juergen Schoenwaelder Jacobs University Bremen gGmbH Phone: +49 421 200 3587 Campus Ring 1, 28759 Bremen, Germany Fax: +49 421 200 3103 From steve@idrathernotsay.com Thu Nov 14 06:50:47 2013 Return-Path: X-Original-To: lmap@ietfa.amsl.com Delivered-To: lmap@ietfa.amsl.com Received: from localhost (localhost [127.0.0.1]) by ietfa.amsl.com (Postfix) with ESMTP id 018B021E80DD for ; Thu, 14 Nov 2013 06:50:47 -0800 (PST) X-Virus-Scanned: amavisd-new at amsl.com X-Spam-Flag: NO X-Spam-Score: -2.197 X-Spam-Level: X-Spam-Status: No, score=-2.197 tagged_above=-999 required=5 tests=[AWL=0.402, BAYES_00=-2.599] Received: from mail.ietf.org ([12.22.58.30]) by localhost (ietfa.amsl.com [127.0.0.1]) (amavisd-new, port 10024) with ESMTP id rmVRhcB77NXY for ; Thu, 14 Nov 2013 06:50:42 -0800 (PST) Received: from colo3.idrathernotsay.com (colo3.idrathernotsay.com [74.207.251.199]) by ietfa.amsl.com (Postfix) with ESMTP id 9050B11E80F2 for ; Thu, 14 Nov 2013 06:50:42 -0800 (PST) Received: from localhost (localhost [127.0.0.1]) by colo3.idrathernotsay.com (Postfix) with ESMTP id 194C9FC4E; Thu, 14 Nov 2013 14:50:42 +0000 (UTC) X-Virus-Scanned: amavisd-new at idrathernotsay.com Received: from colo3.idrathernotsay.com ([127.0.0.1]) by localhost (colo3.idrathernotsay.com [127.0.0.1]) (amavisd-new, port 10024) with ESMTP id u0KUJQ0Emoyj; Thu, 14 Nov 2013 14:50:39 +0000 (UTC) Received: from newbie.idrathernotsay.com (pool-173-79-225-215.washdc.fios.verizon.net [173.79.225.215]) by colo3.idrathernotsay.com (Postfix) with ESMTPSA id 8C782FC36; Thu, 14 Nov 2013 14:50:39 +0000 (UTC) Received: from newbie.idrathernotsay.com (localhost [127.0.0.1]) by newbie.idrathernotsay.com (Postfix) with ESMTP id C71BA4FBD2F; Thu, 14 Nov 2013 09:50:38 -0500 (EST) X-Virus-Scanned: amavisd-new at idrathernotsay.com Received: from newbie.idrathernotsay.com ([127.0.0.1]) by newbie.idrathernotsay.com (newbie.idrathernotsay.com [127.0.0.1]) (amavisd-new, port 10024) with ESMTP id m-NcUzTK-UgW; Thu, 14 Nov 2013 09:50:34 -0500 (EST) Received: by newbie.idrathernotsay.com (Postfix, from userid 1000) id C7A464FC22A; Thu, 14 Nov 2013 09:50:34 -0500 (EST) Date: Thu, 14 Nov 2013 09:50:34 -0500 From: Steve Miller To: "Bugenhagen, Michael K" Message-ID: <20131114145034.GA51673@idrathernotsay.com> References: <9904FB1B0159DA42B0B887B7FA8119CA1292EAA3@AZ-FFEXMB04.global.avaya.com> <20131111203930.GA770@elstar.local> <20131111214712.GA977@elstar.local> MIME-Version: 1.0 Content-Type: text/plain; charset=us-ascii Content-Disposition: inline In-Reply-To: User-Agent: Mutt/1.5.21 (2010-09-15) Cc: "'lmap@ietf.org'" Subject: Re: [lmap] Comments & opinions on Draft - RE: call for consensus - adoption of draft-burbridge-lmap-information-model as WG document X-BeenThere: lmap@ietf.org X-Mailman-Version: 2.1.12 Precedence: list List-Id: Large Scale Measurement of Access network Performance List-Unsubscribe: , List-Archive: List-Post: List-Help: List-Subscribe: , X-List-Received-Date: Thu, 14 Nov 2013 14:50:47 -0000 I guess I'm not clear on how the LMAP WG can really accomplish this. From what I've seen so far, having a LMAP MA coexist on existing hardware would be no big deal. I certainly understand that cablemodems and home routers tend to be seriously low-end devices but I've implemented this sort of thing on such low-end devices before -- sure, you can implement tests that run the device out of resources, but the sorts of things I'd expect people to do routinely shouldn't come close. I don't see how we can tell someone how to integrate this into whatever hardware and OS they're using, that seems well into the weeds of implementation details. If some company has an existing solution that works well for them and that they've gotten the firmware people to put in the devices that company uses, more power to them, no need to change over in a big hurry. The reasons for changing to using LMAP would seem to be the usual ones: you don't have to have a staff that wrangles the hardware vendor to do what you want (or in my case, a staff that writes the MA code), you don't have to have custom infrastructure (or as much of it, at least) on the collector side, if you're in a regulated industry and your regulator says "use this standard!" you can use it. (-: -Steve On Thu, Nov 14, 2013 at 12:31:22PM +0000, Bugenhagen, Michael K wrote: > Juergen, > > I'm trying to tell you that the Operations groups in the industry are putting probes on the Modems to troubleshoot all the services. > If the new LMAP probe can't co-exist with an existing probe that is deployed everywhere that would be highly problematic for some ISP's. > > My 15+ years of Operations troubleshooting experience is often "asked" for in standards discussions -not always appreciated though.. > > :) > > Hope that clears up why I'm recommending we make sure isn't a complete testing overlay that can't exist with other existing overlays. > In other words - If an ISP has a testing / MA / Probe control plane that's can't co-exist with this new one... one will lose out. > > Thanks, > Mike > > > > -----Original Message----- > From: Juergen Schoenwaelder [mailto:j.schoenwaelder@jacobs-university.de] > Sent: Monday, November 11, 2013 3:47 PM > To: Bugenhagen, Michael K > Cc: 'Romascanu, Dan (Dan)'; lmap@ietf.org; 'Trevor Burbridge'; 'philip.eardley@bt.com'; 'marcelo bagnulo braun'; 'Stark, Barbara'; Cook, Charles; 'MORTON, ALFRED C (AL)'; 'KEN KO' > Subject: Re: [lmap] Comments & opinions on Draft - RE: call for consensus - adoption of draft-burbridge-lmap-information-model as WG document > > On Mon, Nov 11, 2013 at 08:46:35PM +0000, Bugenhagen, Michael K wrote: > > Juergen, > > > > If you look at the bigger picture my comments are saying we should consider those 2 items as requirements of the use cases. > > Or at least consider them as such... before running too far into any model that may break if critical requirements are missed. > > > > The 2 I am suggesting here - > > > > 1) ISP's need a MA that can Stand by itself as an Overlay, but more importantly they may need one that operates more globally as a universal MA. > > 2) If we adopt a registry and we know virtualization is on it's way, then we need to take that into account (a virtual MA). > > > > I am still trying to understand what you are suggesting. > > - What does "ISP's need a MA that can Stand by itself as an Overlay" > mean? > > - What does "ISP's ... need one that operates more globally as a > universal MA" mean? What is a universal MA? What is more globally > here? > > - What registry are you talking about? It seems you are not talking > about the IPPM metrics registry. So what do you have in mind? > > Reading again your other email, if you are talking about an MA under control of multiple controllers, then I am afraid LMAP 1.0 won't address this (see the WG's charter). But MAs can of course implement tests for multiple metrics. > > /js > > -- > Juergen Schoenwaelder Jacobs University Bremen gGmbH > Phone: +49 421 200 3587 Campus Ring 1, 28759 Bremen, Germany > Fax: +49 421 200 3103 > _______________________________________________ > lmap mailing list > lmap@ietf.org > https://www.ietf.org/mailman/listinfo/lmap From jason.weil@twcable.com Thu Nov 14 08:59:47 2013 Return-Path: X-Original-To: lmap@ietfa.amsl.com Delivered-To: lmap@ietfa.amsl.com Received: from localhost (localhost [127.0.0.1]) by ietfa.amsl.com (Postfix) with ESMTP id 6960E21E8098 for ; Thu, 14 Nov 2013 08:59:47 -0800 (PST) X-Virus-Scanned: amavisd-new at amsl.com X-Spam-Flag: NO X-Spam-Score: 0.5 X-Spam-Level: X-Spam-Status: No, score=0.5 tagged_above=-999 required=5 tests=[AWL=-0.897, BAYES_20=-0.74, HELO_EQ_MODEMCABLE=0.768, HOST_EQ_MODEMCABLE=1.368, HTML_MESSAGE=0.001] Received: from mail.ietf.org ([12.22.58.30]) by localhost (ietfa.amsl.com [127.0.0.1]) (amavisd-new, port 10024) with ESMTP id YOP9pv5YHRX8 for ; Thu, 14 Nov 2013 08:59:43 -0800 (PST) Received: from cdcipgw01.twcable.com (cdcipgw01.twcable.com [165.237.91.110]) by ietfa.amsl.com (Postfix) with ESMTP id 93CA821E8056 for ; Thu, 14 Nov 2013 08:59:42 -0800 (PST) X-SENDER-IP: 10.136.163.11 X-SENDER-REPUTATION: None X-IronPort-AV: E=Sophos;i="4.93,700,1378872000"; d="scan'208,217";a="55809958" Received: from unknown (HELO PRVPEXHUB02.corp.twcable.com) ([10.136.163.11]) by cdcipgw01.twcable.com with ESMTP/TLS/RC4-MD5; 14 Nov 2013 11:59:07 -0500 Received: from PRVPEXVS17.corp.twcable.com ([10.136.163.95]) by PRVPEXHUB02.corp.twcable.com ([10.136.163.11]) with mapi; Thu, 14 Nov 2013 11:59:41 -0500 From: "Weil, Jason" To: "lmap@ietf.org" Date: Thu, 14 Nov 2013 11:59:40 -0500 Thread-Topic: IETF88 - LMAP WG Meeting Notes Posted Thread-Index: Ac7hWuj+aVDCh98uSgONIw9oUM2B9w== Message-ID: Accept-Language: en-US Content-Language: en-US X-MS-Has-Attach: X-MS-TNEF-Correlator: user-agent: Microsoft-MacOutlook/14.3.8.130913 acceptlanguage: en-US Content-Type: multipart/alternative; boundary="_000_CEAA6BAC218B9jasonweiltwcablecom_" MIME-Version: 1.0 Subject: [lmap] IETF88 - LMAP WG Meeting Notes Posted X-BeenThere: lmap@ietf.org X-Mailman-Version: 2.1.12 Precedence: list List-Id: Large Scale Measurement of Access network Performance List-Unsubscribe: , List-Archive: List-Post: List-Help: List-Subscribe: , X-List-Received-Date: Thu, 14 Nov 2013 16:59:47 -0000 --_000_CEAA6BAC218B9jasonweiltwcablecom_ Content-Type: text/plain; charset="us-ascii" Content-Transfer-Encoding: quoted-printable LMAP WG, The meeting minutes from the IETF 88 WG session have been posted to the dat= atracker site. The minutes can be found at the following link: http://www.ietf.org/proceedings/88/minutes/minutes-88-lmap I would like to thank our note takers Sam Aldrin and Steve Miller for their= thorough notes capturing the questions and comments made during the Vancou= ver LMAP WG session. Please take a minute and review the notes and provide= any feedback regarding additions or corrections that need to be addressed = prior to the final proceedings cutoff date of December 6, 2013. Sincerely, Jason and Dan ________________________________ This E-mail and any of its attachments may contain Time Warner Cable propri= etary information, which is privileged, confidential, or subject to copyrig= ht belonging to Time Warner Cable. This E-mail is intended solely for the u= se of the individual or entity to which it is addressed. If you are not the= intended recipient of this E-mail, you are hereby notified that any dissem= ination, distribution, copying, or action taken in relation to the contents= of and attachments to this E-mail is strictly prohibited and may be unlawf= ul. If you have received this E-mail in error, please notify the sender imm= ediately and permanently delete the original and any copy of this E-mail an= d any printout. --_000_CEAA6BAC218B9jasonweiltwcablecom_ Content-Type: text/html; charset="us-ascii" Content-Transfer-Encoding: quoted-printable
LMAP WG,

The meeting minutes from the IETF 88 WG session have been posted to th= e datatracker site. The minutes can be found at the following link:
 

I would like to thank our note takers Sam Aldrin and Steve Miller for = their thorough notes capturing the questions and comments made during the V= ancouver LMAP WG session.  Please take a minute and review the notes a= nd provide any feedback regarding additions or corrections that need to be addressed prior to the final proceedings cu= toff date of December 6, 2013.  

Sincerely, 

Jason and Dan



This E-mail and any of its a= ttachments may contain Time Warner Cable proprietary information, which is = privileged, confidential, or subject to copyright belonging to Time Warner = Cable. This E-mail is intended solely for the use of the individual or entity to which it is addressed. If you a= re not the intended recipient of this E-mail, you are hereby notified that = any dissemination, distribution, copying, or action taken in relation to th= e contents of and attachments to this E-mail is strictly prohibited and may be unlawful. If you have receiv= ed this E-mail in error, please notify the sender immediately and permanent= ly delete the original and any copy of this E-mail and any printout.
 --_000_CEAA6BAC218B9jasonweiltwcablecom_-- From dromasca@avaya.com Mon Nov 18 06:54:44 2013 Return-Path: X-Original-To: lmap@ietfa.amsl.com Delivered-To: lmap@ietfa.amsl.com Received: from localhost (localhost [127.0.0.1]) by ietfa.amsl.com (Postfix) with ESMTP id 13E2111E8142 for ; Mon, 18 Nov 2013 06:54:44 -0800 (PST) X-Virus-Scanned: amavisd-new at amsl.com X-Spam-Flag: NO X-Spam-Score: -103.599 X-Spam-Level: X-Spam-Status: No, score=-103.599 tagged_above=-999 required=5 tests=[BAYES_00=-2.599, RCVD_IN_DNSWL_LOW=-1, USER_IN_WHITELIST=-100] Received: from mail.ietf.org ([12.22.58.30]) by localhost (ietfa.amsl.com [127.0.0.1]) (amavisd-new, port 10024) with ESMTP id yJbRgi0zWmbd for ; Mon, 18 Nov 2013 06:54:37 -0800 (PST) Received: from de307622-de-outbound.net.avaya.com (de307622-de-outbound.net.avaya.com [198.152.71.100]) by ietfa.amsl.com (Postfix) with ESMTP id EB09B11E818D for ; Mon, 18 Nov 2013 06:50:44 -0800 (PST) X-IronPort-Anti-Spam-Filtered: true X-IronPort-Anti-Spam-Result: AmcGAMEoilKHCzI1/2dsb2JhbABZgmYhOFOCdbxBGIECFm0HgiUBAQEBAxILBhFRBgEIDQQEAQEDAgYLAQEBDwMCBDAUAQYBAQUFBBMIGodfAQyhH4RIikCSExMEgSmMdBACAYEIPgQIAQWCUzWBEQOULopIiyeDKIFqBxci X-IronPort-AV: E=Sophos;i="4.93,724,1378872000"; d="scan'208";a="32134408" Received: from unknown (HELO p-us1-erheast.us1.avaya.com) ([135.11.50.53]) by de307622-de-outbound.net.avaya.com with ESMTP; 18 Nov 2013 09:50:42 -0500 Received: from unknown (HELO AZ-FFEXHC03.global.avaya.com) ([135.64.58.13]) by p-us1-erheast-out.us1.avaya.com with ESMTP; 18 Nov 2013 09:40:19 -0500 Received: from AZ-FFEXMB04.global.avaya.com ([fe80::6db7:b0af:8480:c126]) by AZ-FFEXHC03.global.avaya.com ([135.64.58.13]) with mapi id 14.03.0146.000; Mon, 18 Nov 2013 09:50:40 -0500 From: "Romascanu, Dan (Dan)" To: "lmap@ietf.org" Thread-Topic: NOMCOM - final call for feedback, nominee list Thread-Index: AQHO4wEQO2eu2ch9Uk+OukcaxgM/aporFCWQ Date: Mon, 18 Nov 2013 14:50:39 +0000 Message-ID: <9904FB1B0159DA42B0B887B7FA8119CA1293F84A@AZ-FFEXMB04.global.avaya.com> Accept-Language: en-US Content-Language: en-US X-MS-Has-Attach: X-MS-TNEF-Correlator: x-originating-ip: [135.64.58.46] Content-Type: text/plain; charset="utf-8" Content-Transfer-Encoding: base64 MIME-Version: 1.0 Subject: [lmap] FW: NOMCOM - final call for feedback, nominee list X-BeenThere: lmap@ietf.org X-Mailman-Version: 2.1.12 Precedence: list List-Id: Large Scale Measurement of Access network Performance List-Unsubscribe: , List-Archive: List-Post: List-Help: List-Subscribe: , X-List-Received-Date: Mon, 18 Nov 2013 14:54:44 -0000 UGxlYXNlIHNlZSBiZWxvdyB0aGUgbWVzc2FnZSBmcm9tIHRoZSBOb21jb20gY2hhaXIuIFBsZWFz ZSBoZWxwIE5vbWNvbSB3aXRoIHlvdXIgZmVlZGJhY2sgLSB0aGlzIGlzIGFuIGltcG9ydGFudCBj b250cmlidXRpb24gb2YgaGlnaCBpbXBvcnRhbmNlIGZvciBOb21jb20gaW4gdGhlaXIgdGFzayBv ZiBzZWxlY3RpbmcgdGhlIGJlc3QgbmV3IGxlYWRlcnMgb3V0IG9mIHRoZSBwb29sIG9mIHRhbGVu dGVkIGNhbmRpZGF0ZXMuIA0KDQpUaGFua3MgYW5kIFJlZ2FyZHMsDQoNCkRhbg0KDQoNCg0KDQot LS0tLU9yaWdpbmFsIE1lc3NhZ2UtLS0tLQ0KRnJvbTogaWV0Zi1hbm5vdW5jZS1ib3VuY2VzQGll dGYub3JnIFttYWlsdG86aWV0Zi1hbm5vdW5jZS1ib3VuY2VzQGlldGYub3JnXSBPbiBCZWhhbGYg T2YgTm9tQ29tIENoYWlyIDIwMTMNClNlbnQ6IFNhdHVyZGF5LCBOb3ZlbWJlciAxNiwgMjAxMyA5 OjIxIFBNDQpUbzogSUVURiBBbm5vdW5jZW1lbnQgTGlzdA0KQ2M6IHdnY2hhaXJzQGlldGYub3Jn DQpTdWJqZWN0OiBOT01DT00gLSBmaW5hbCBjYWxsIGZvciBmZWVkYmFjaywgbm9taW5lZSBsaXN0 DQoNCkhlbHAgdGhlIE5vbWNvbTogIHRoaXMgaXMgdGhlIGxhc3QgY2FsbCBmb3IgZ2VuZXJhbCBm ZWVkYmFjayBvbiBub21pbmVlcy4gVGhpcyBlbWFpbCBpbmNsdWRlcyB0aGUgd2lsbGluZyBub21p bmVlIGxpc3QgZm9yIHlvdXIgY29udmVuaWVuY2UsIGFzIHBlciBSRkMgNTY4MC4gV0cgY2hhaXJz LCBwbGVhc2UgZm9yd2FyZCB0aGlzIGVtYWlsIHRvIHlvdXIgV0dzLg0KDQpXQVJOSU5HIC0gRE8g Tk9UIHJlcGx5IHRvIHRoaXMgZW1haWwgd2l0aCBmZWVkYmFjayAtIHRoZSBhbm5vdW5jZW1lbnQg dG9vbCBwcm92aWRlcyBhbiB1bnByZWRpY3RhYmxlIFJlcGx5LVRvIGZpZWxkLiANCllvdSBtYXkg c2VuZCBtYWlsIHRvIG5vbWNvbTEzLCBidXQgcGxlYXNlIGRvIGNvbnRpbnVlIHJlYWRpbmcgZm9y IGFkZGl0aW9uYWwgaW5mby4NCg0KVGhlIGRlYWRsaW5lIGlzIG1pZG5pZ2h0IChhbnkgdGltZXpv bmUpIFdlZCBOb3YgMjAuDQoNCkVudGVyIHlvdXIgZmVlZGJhY2sgaW50byB0aGUgZW5jcnlwdGVk IG5vbWNvbSB0b29sIGF0Og0KDQpodHRwczovL2RhdGF0cmFja2VyLmlldGYub3JnL25vbWNvbS8y MDEzL2ZlZWRiYWNrLw0KDQpBbGwgZmVlZGJhY2sgaXMgY29uZmlkZW50aWFsLg0KDQpTZWxlY3Qg aW5kaXZpZHVhbCBub21pbmVlcywgb3Igc2VsZWN0IGFuIGFyZWEgKGUuZy4gSU5UIE11bHRpcGxl KSBpZiB5b3Ugd2FudCB0byBnaXZlIGZlZWRiYWNrIG9uIG11bHRpcGxlIG5vbWluZWVzIGZvciB0 aGF0IGFyZWEgaW4gb25lIGVudHJ5Lg0KDQpUaGUgTm9tY29tIHdpbGwgZ3JlYXRseSBhcHByZWNp YXRlIHlvdXIgY29uY3JldGUgYW5kIGRldGFpbGVkIHRob3VnaHRzIG9uIG5vbWluZWVzLiAgDQoN CllvdSBuZWVkIHRvIGhhdmUgYSBkYXRhdHJhY2tlciBhY2NvdW50IGluIG9yZGVyIHRvIGVudGVy IGZlZWRiYWNrLiAgDQpUaGlzIGlzIHRoZSBzaXRlIHRvIGFjY2VzcyBlaXRoZXIgdG8gY3JlYXRl IGEgZGF0YXRyYWNrZXIgYWNjb3VudCBvciB0byByZXNldCB0aGUgcGFzc3dvcmQgb24gYW4gZXhp c3RpbmcgYWNjb3VudDoNCg0KaHR0cHM6Ly9kYXRhdHJhY2tlci5pZXRmLm9yZy9hY2NvdW50cy8N Cg0KQXMgd2UgYW5ub3VuY2VkIGxhc3Qgd2Vlaywgd2UgYXJlIHNoYXJpbmcgdGhlIG5hbWVzIG9m IHRoZSB3aWxsaW5nIG5vbWluZWVzIGluIHRoaXMgZW1haWwsIGFzIHBhcnQgb2Ygb3VyIHVzZSBv ZiBSRkMgNTY4MCBPcGVuIExpc3QgaW4gdGhpcyBub21jb20gY3ljbGUuICBUaGVzZSBuYW1lcyAo YXBwZWFyaW5nIGF0IHRoZSBlbmQgb2YgdGhlIGVtYWlsKSBoYXZlIGJlZW4gYXZhaWxhYmxlIHdp dGhpbiB0aGUgZGF0YXRyYWNrZXIgc2luY2Ugb3VyIGZpcnN0IGNhbGwgZm9yIGZlZWRiYWNrOyB3 ZSBob3BlIHNvbWUgcGVvcGxlIHdpbGwgYmUgZXh0cmEgZW5jb3VyYWdlZCB0byBwcm92aWRlDQpm ZWVkYmFjayBieSBzZWVpbmcgdGhlIG5hbWVzIHByaW9yIHRvIGRhdGF0cmFja2VyIGxvZ2luLiAg IEFsbCB5b3VyDQpmZWVkYmFjayBtdXN0IGJlIGVudGVyZWQgaW4gdGhlIGVuY3J5cHRlZCBub21j b20gc2l0ZSAob3IgbWFpbGVkIHRvIG5vbWNvbS1jaGFpci0yMDEzQGlldGYub3JnIG9yIG5vbWNv bTEzQGlldGYub3JnKS4gIFlvdSBtYXkgYXNrIHRoZSBjaGFpciBvciBhbnkgYW5vdGhlciBub21j b20gbWVtYmVyIHRvIHN1Ym1pdCB5b3VyIGZlZWRiYWNrIHdpdGhvdXQgeW91ciBuYW1lIGF0dGFj aGVkLCBpZiB5b3Ugd291bGQgcHJlZmVyLg0KDQpBZ2FpbiwgeW91ciBmZWVkYmFjayBpcyBhYnNv bHV0ZWx5IGNvbmZpZGVudGlhbCwgaG93ZXZlciBkZWxpdmVyZWQuDQoNClRoYW5rcyB2ZXJ5IG11 Y2ggZnJvbSBhbGwgb2YgdXMgZm9yIHlvdXIgdGltZSBhbmQgZm9yIGhlbHBpbmcgdGhlIE5vbWNv bSB0byBkbyB0aGUgYmVzdCBwb3NzaWJsZSBqb2IuICBBbHNvIGVub3Jtb3VzIHRoYW5rcyB0byB0 aGUgd2lsbGluZyBub21pbmVlcy4NCg0KTmFtZXMgYmVsb3cuDQoNCkFsbGlzb24gZm9yIHRoZSBO b21jb20NCg0KQVBQDQpKb2UgSGlsZGVicmFuZA0KQmFycnkgTGVpYmEgKGluY3VtYmVudCkNCllv c2hpcm8gWW9uZXlhDQoNCklOVA0KRG9uYWxkIEVhc3RsYWtlDQpXZXNsZXkgR2VvcmdlDQpCcmlh biBIYWJlcm1hbiAoaW5jdW1iZW50KQ0KT2xlIFRyb2FuDQoNCk9QUw0KQmVub2l0IENsYWlzZSAo aW5jdW1iZW50KQ0KTGluZGEgRHVuYmFyDQpWaWN0b3IgS3VhcnNpbmdoDQpCaWxsIE1hbm5pbmcN Cg0KUkFJDQpCZW4gQ2FtcGJlbGwNCkFsaXNzYSBDb29wZXINCktlaXRoIERyYWdlDQpEYW4gUm9t YXNjYW51DQoNClJURw0KTG9hIEFuZGVyc3Nvbg0KQWxpYSBBdGxhcw0KRGVib3JhaCBCcnVuZ2Fy ZA0KU3Rld2FydCBCcnlhbnQgKGluY3VtYmVudCkNCkRvbmFsZCBFYXN0bGFrZQ0KU3VzYW4gSGFy ZXMNCktleXVyIFBhdGVsDQoNClNFQw0KRGVyZWsgQXRraW5zDQpEb25hbGQgRWFzdGxha2UNCkpl ZmYgSG9kZ2VzDQpMZWlmIEpvaGFuc3Nvbg0KQWxleGV5IE1lbG5pa292DQpLYXRobGVlbiBNb3Jp YXJ0eQ0KRXJpYyBSZXNjb3JsYQ0KDQpUU1YNCk1hcnRpbiBTdGllbWVybGluZyAoaW5jdW1iZW50 KQ0KRGF2ZSBUYWh0DQoNCklBQg0KQmVybmFyZCBBYm9iYSAoaW5jdW1iZW50KQ0KTG9hIEFuZGVy c3Nvbg0KQWxpYXMgQXRsYXMNCk1hcnkgQmFybmVzDQpNYXJjIEJsYW5jaGV0IChpbmN1bWJlbnQp DQpSb3NzIENhbGxvbiAoaW5jdW1iZW50KQ0KSGFnbyBEYWZhbGxhDQpMaW5kYSBEdW5iYXINClJv bmkgRXZlbg0KSmltIEdldHR5cw0KVGVkIEhhcmRpZQ0KU3VzYW4gSGFyZXMNCkpvZSBIaWxkZWJy YW5kDQpTaGVuZyBKaWFuZw0KRWxpb3QgTGVhciAoaW5jdW1iZW50KQ0KTGFycnkgTWFzaW50ZXIN ClMuIE1vb25lc2FteQ0KS2F0aGxlZW4gTW9yaWFydHkNCkthdmVoIFJhbmpiYXINClJvYmVydCBT cGFya3MNClRpbmEgVHNvdQ0KQnJpYW4gVHJhbW1lbGwNCg0KSUFPQy9JRVRGIFRydXN0DQpHbGVu biBEZWVuDQpUb2JpYXMgR29uZHJvbQ0KQ2hyaXMgR3JpZmZpdGhzIChpbmN1bWJlbnQpDQpKb2hu IExldmluZQ0KVGluYSBUc291DQoNCg0KDQoNCg0KDQoNCg0KDQoNCg0KDQoNCg0KDQoNCg0KDQoN Cg0KDQoNCg0KDQoNCg0KDQoNCg0KDQoNCg== From dromasca@avaya.com Thu Nov 21 04:30:32 2013 Return-Path: X-Original-To: lmap@ietfa.amsl.com Delivered-To: lmap@ietfa.amsl.com Received: from localhost (ietfa.amsl.com [127.0.0.1]) by ietfa.amsl.com (Postfix) with ESMTP id 4BD211ADEBE for ; Thu, 21 Nov 2013 04:30:32 -0800 (PST) X-Virus-Scanned: amavisd-new at amsl.com X-Spam-Flag: NO X-Spam-Score: -2.425 X-Spam-Level: X-Spam-Status: No, score=-2.425 tagged_above=-999 required=5 tests=[BAYES_00=-1.9, RP_MATCHES_RCVD=-0.525] autolearn=ham Received: from mail.ietf.org ([4.31.198.44]) by localhost (ietfa.amsl.com [127.0.0.1]) (amavisd-new, port 10024) with ESMTP id EDBn4v0IbjDW for ; Thu, 21 Nov 2013 04:30:31 -0800 (PST) Received: from p-us1-iereast-outbound.us1.avaya.com (p-us1-iereast-outbound.us1.avaya.com [135.11.29.13]) by ietfa.amsl.com (Postfix) with ESMTP id 2C2E11ADBD4 for ; Thu, 21 Nov 2013 04:30:31 -0800 (PST) X-IronPort-Anti-Spam-Filtered: true X-IronPort-Anti-Spam-Result: AvcFACr8jVLGmAcV/2dsb2JhbABZgmYhgQu8fIEhFm0HgicBAQMSCx1RARUVFEImAQQbGodfAaBYhEibV44wAQGBCINYgRIDnnuDb4c4gyiBcTk X-IronPort-AV: E=Sophos;i="4.93,743,1378872000"; d="scan'208";a="38087242" Received: from unknown (HELO co300216-co-erhwest-exch.avaya.com) ([198.152.7.21]) by p-us1-iereast-outbound.us1.avaya.com with ESMTP; 21 Nov 2013 07:30:24 -0500 Received: from unknown (HELO AZ-FFEXHC01.global.avaya.com) ([135.64.58.11]) by co300216-co-erhwest-out.avaya.com with ESMTP; 21 Nov 2013 07:23:57 -0500 Received: from AZ-FFEXMB04.global.avaya.com ([fe80::6db7:b0af:8480:c126]) by AZ-FFEXHC01.global.avaya.com ([135.64.58.11]) with mapi id 14.03.0146.000; Thu, 21 Nov 2013 13:30:22 +0100 From: "Romascanu, Dan (Dan)" To: "lmap@ietf.org" Thread-Topic: consensus on adopting draft-burbridge-lmap-information-model as WG document Thread-Index: Ac7mtXIafNqyJWO2QqCAVThK1VbXDw== Date: Thu, 21 Nov 2013 12:30:22 +0000 Message-ID: <9904FB1B0159DA42B0B887B7FA8119CA12943A77@AZ-FFEXMB04.global.avaya.com> Accept-Language: en-US Content-Language: en-US X-MS-Has-Attach: X-MS-TNEF-Correlator: x-originating-ip: [135.64.58.46] Content-Type: text/plain; charset="us-ascii" Content-Transfer-Encoding: quoted-printable MIME-Version: 1.0 Subject: [lmap] consensus on adopting draft-burbridge-lmap-information-model as WG document X-BeenThere: lmap@ietf.org X-Mailman-Version: 2.1.15 Precedence: list List-Id: Large Scale Measurement of Access network Performance List-Unsubscribe: , List-Archive: List-Post: List-Help: List-Subscribe: , X-List-Received-Date: Thu, 21 Nov 2013 12:30:32 -0000 Following the discussions and consensus in the room at IETF-88 we initiated= a call for consensus on the mail list on adopting draft-burbridge-lmap-inf= ormation-model as a WG document. This call generated interesting discussion= s on the WG mail list, but none of the comments expressed disapproval, neit= her other contributions were submitted for the LMAP information model. As c= hairs we consider that the WG reached consensus on this matter and we instr= uct the document authors to submit the next version of the document as draf= t-lmap-information-model-00.txt. Thanks and Regards, Jason and Dan From philip.eardley@bt.com Mon Nov 25 08:55:30 2013 Return-Path: X-Original-To: lmap@ietfa.amsl.com Delivered-To: lmap@ietfa.amsl.com Received: from localhost (ietfa.amsl.com [127.0.0.1]) by ietfa.amsl.com (Postfix) with ESMTP id 0868C1ADF54 for ; Mon, 25 Nov 2013 08:55:30 -0800 (PST) X-Virus-Scanned: amavisd-new at amsl.com X-Spam-Flag: NO X-Spam-Score: -1.935 X-Spam-Level: X-Spam-Status: No, score=-1.935 tagged_above=-999 required=5 tests=[BAYES_00=-1.9, RCVD_IN_DNSWL_LOW=-0.7, SPF_SOFTFAIL=0.665] autolearn=no Received: from mail.ietf.org ([4.31.198.44]) by localhost (ietfa.amsl.com [127.0.0.1]) (amavisd-new, port 10024) with ESMTP id PyyIbm9Ltk01 for ; Mon, 25 Nov 2013 08:55:28 -0800 (PST) Received: from smtpe1.intersmtp.com (smtp64.intersmtp.com [62.239.224.237]) by ietfa.amsl.com (Postfix) with ESMTP id B380B1AD947 for ; Mon, 25 Nov 2013 08:55:27 -0800 (PST) Received: from EVMHT65-UKRD.domain1.systemhost.net (10.36.3.102) by RDW083A008ED64.smtp-e4.hygiene.service (10.187.98.13) with Microsoft SMTP Server (TLS) id 8.3.298.1; Mon, 25 Nov 2013 16:55:26 +0000 Received: from EMV67-UKRD.domain1.systemhost.net ([169.254.1.152]) by EVMHT65-UKRD.domain1.systemhost.net ([10.36.3.102]) with mapi; Mon, 25 Nov 2013 16:55:26 +0000 From: To: , Date: Mon, 25 Nov 2013 16:55:24 +0000 Thread-Topic: [lmap] Comments on the current round of I-Ds, before the meeting tomorrow Thread-Index: Ac7aplg2ABTnDip0SbKDaRXBk9X57wPWGlrA Message-ID: References: <20131106041126.GA57287@idrathernotsay.com> In-Reply-To: <20131106041126.GA57287@idrathernotsay.com> Accept-Language: en-US, en-GB Content-Language: en-US X-MS-Has-Attach: X-MS-TNEF-Correlator: acceptlanguage: en-US, en-GB Content-Type: text/plain; charset="us-ascii" Content-Transfer-Encoding: quoted-printable MIME-Version: 1.0 Subject: Re: [lmap] Comments on the current round of I-Ds, before the meeting tomorrow X-BeenThere: lmap@ietf.org X-Mailman-Version: 2.1.15 Precedence: list List-Id: Large Scale Measurement of Access network Performance List-Unsubscribe: , List-Archive: List-Post: List-Help: List-Subscribe: , X-List-Received-Date: Mon, 25 Nov 2013 16:55:30 -0000 Steve Thanks for all the comments Replying first on use cases > -----Original Message----- > From: lmap-bounces@ietf.org [mailto:lmap-bounces@ietf.org] On Behalf Of > Steve Miller > Sent: 06 November 2013 04:11 > To: lmap@ietf.org > Subject: [lmap] Comments on the current round of I-Ds, before the > meeting tomorrow >=20 ....=20 > LMAP Use Cases (draft-ietf-lmap-use-cases-00) > --------------------------------------------- > There's a specific use case in which I have quite a lot of interest, > and while the text that's here already certainly doesn't preclude that > use case, I don't know if it'd be worth laying out the use case I > envision, or if it's too special-case. >=20 > I think there's an unspoken assumption here that a measurement is from > point A to point B, when for certain types of measurements -- DNS being > the first one that comes to my mind! (-: -- are from point A to one of > potentially many points B, because B is anycasted. That's also not too > different from the case where A finds B via a DNS lookup, but B has > more than one IP address. >=20 > Most, if not all, of [a-m].root-servers.net are anycasted, as are > several of the com/net service addresses, and certainly Verisign's > peers and competitors do the same sorts of things, they're standard > operating procedure in the DNS world. >=20 > Something that I think all operators have to deal with is that with > heavily anycasted infrastructure, it's easy to end up in a situation > where some users have poor connectivity to some service instance, but > there's no good way for the operator of the infrastructure to *know* > that some users are having issues. For example, if someone connected > to an ISP in Cairo is having issues hitting the j.root-servers.net > instance in Cairo, I don't have a good way to know that unless I can > also test from within that user's ISP. >=20 > So having the ability to perform a small number of tests from a very > broad range of MAs with wide IP-topologic scope would be massively > useful. Going along with that, we'd need to have some way as part of > the measurement being performed to have the MA identify the instance > it's reached -- including the result of a "chaos null hostname.bind" > query, a traceroute, or somesuch, would be needed. >=20 > Anyway, we might be weird, but since I think we have a lot of company, > I thought it was worth calling out this use case... >=20 > There's also a lot of emphasis in this and the other docs about backing > off when the link is in use, but I think there's a lot to be learned > from performing relatively low-bandwidth tests all the time, even when > the link is otherwise in use, small enough that no real backoff > approach is needed. You can learn a lot from pinging something every > 10 seconds or so, without burning a lot of bandwidth. I agree that at > least in certain scenarios we'd have to worry about counting against > someone's data allowance. I think this could be worth mentioning more directly in the use cases &/or = framework. Will add something in the next iteration Will also mention about the value of low bandwidth tests, completely agree = with you on this Thanks phil From philip.eardley@bt.com Tue Nov 26 01:52:49 2013 Return-Path: X-Original-To: lmap@ietfa.amsl.com Delivered-To: lmap@ietfa.amsl.com Received: from localhost (ietfa.amsl.com [127.0.0.1]) by ietfa.amsl.com (Postfix) with ESMTP id 15FE81AE125 for ; Tue, 26 Nov 2013 01:52:49 -0800 (PST) X-Virus-Scanned: amavisd-new at amsl.com X-Spam-Flag: NO X-Spam-Score: -1.334 X-Spam-Level: X-Spam-Status: No, score=-1.334 tagged_above=-999 required=5 tests=[BAYES_00=-1.9, HTML_MESSAGE=0.001, J_CHICKENPOX_72=0.6, RCVD_IN_DNSWL_LOW=-0.7, SPF_SOFTFAIL=0.665] autolearn=no Received: from mail.ietf.org ([4.31.198.44]) by localhost (ietfa.amsl.com [127.0.0.1]) (amavisd-new, port 10024) with ESMTP id 7q538s36qDKN for ; Tue, 26 Nov 2013 01:52:45 -0800 (PST) Received: from smtpe1.intersmtp.com (smtp63.intersmtp.com [62.239.224.236]) by ietfa.amsl.com (Postfix) with ESMTP id 2A3A41A1F55 for ; Tue, 26 Nov 2013 01:52:44 -0800 (PST) Received: from EVMHT66-UKRD.domain1.systemhost.net (10.36.3.103) by RDW083A007ED63.smtp-e3.hygiene.service (10.187.98.12) with Microsoft SMTP Server (TLS) id 8.3.298.1; Tue, 26 Nov 2013 09:52:43 +0000 Received: from EMV67-UKRD.domain1.systemhost.net ([169.254.1.152]) by EVMHT66-UKRD.domain1.systemhost.net ([10.36.3.103]) with mapi; Tue, 26 Nov 2013 09:52:43 +0000 From: To: , Date: Tue, 26 Nov 2013 09:52:42 +0000 Thread-Topic: [lmap] LMAP regulator use case Thread-Index: Ac7avCdjLo9Im2hHQqCbAgHsZO9kAwPz99+w Message-ID: References: <793D91975B99224DA9777562FF192808A27468CB@exmbx01> <6AF4522BD5AB86429CFD3948A9AB2F2F3A22854A@loota.laru.local> In-Reply-To: Accept-Language: en-US, en-GB Content-Language: en-US X-MS-Has-Attach: X-MS-TNEF-Correlator: acceptlanguage: en-US, en-GB Content-Type: multipart/alternative; boundary="_000_A2E337CDB7BC4145B018B9BEE8EB3E0D403B78C747EMV67UKRDdoma_" MIME-Version: 1.0 Cc: hgs@cs.columbia.edu, mlinsner@cisco.com, lmap@ietf.org, frode.sorensen@npt.no Subject: Re: [lmap] LMAP regulator use case X-BeenThere: lmap@ietf.org X-Mailman-Version: 2.1.15 Precedence: list List-Id: Large Scale Measurement of Access network Performance List-Unsubscribe: , List-Archive: List-Post: List-Help: List-Subscribe: , X-List-Received-Date: Tue, 26 Nov 2013 09:52:49 -0000 --_000_A2E337CDB7BC4145B018B9BEE8EB3E0D403B78C747EMV67UKRDdoma_ Content-Type: text/plain; charset="us-ascii" Content-Transfer-Encoding: quoted-printable James, I agree that privacy (of the end user who has the MA) is important - in the= ISP use case as well as the regulator one http://tools.ietf.org/html/draft-ietf-lmap-framework-01#section-8 discusses= privacy (data protection). It would be great to get your feedback on this = section. One idea that's been raised is a Group-ID (section 8.7) - would be= very interested to hear your opinion and also other ideas for improving pr= ivacy (personally am not convinced by the Group-ID, as seems a direct trade= -off with security) thanks phil From: James Miller [mailto:jamesmilleresquire@gmail.com] Sent: 06 November 2013 06:47 To: Nieminen Klaus Cc: Eardley,PL,Philip,TUB8 R; frode.sorensen@npt.no; lmap@ietf.org; Marc Li= nsner; Henning Schulzrinne Subject: Re: [lmap] LMAP regulator use case I would propose adding in section 2 or 4 some text describing a regulators = need to ensure that policies intended to protect the privacy interests of a= potential tester can be enforced using features available in the system. = Recognizing that http://www.ietf.org/id/draft-folks-lmap-framework-00.txt a= nd in the lists there's a view that a single entity will be overseeing a la= rge scale measurement, there is a clear need for a test infrastructure to i= nclude tools to enforce the privacy policies of the managing entity and coo= rdinate collaborative activity. A desire to protect the privacy of testers= contributing to our mobile measurement effort lead our work on the project= , and our decision to adopt changes to our fixed technical infrastructure a= nd data model to support a totally anonymous collection at the MA and commu= nications between controllers, and through the collectors and the test resu= lts collected. I understand there has been discussion on this point in the= past and we had included this in the Section 7 Security Considerations of = our original I-D. In any event I think the regulator use case benefits fro= m calling the point out specifically. http://tools.ietf.org/id/draft-schulzrinne-lmap-requirements-00.txt Proposed addition in Section 2.2: :: Regulators role in the development and enforcement of broadband Internet= access service policies also require that the measurement approaches meet = a high level of verifiability, accuracy and provider-independence to suppor= t valid and meaningful comparisons of Internet access service performance. = Regulators also require that policies intended to protect the privacy int= erests of potential testers can be enforced using features available in the= system. ... --_000_A2E337CDB7BC4145B018B9BEE8EB3E0D403B78C747EMV67UKRDdoma_ Content-Type: text/html; charset="us-ascii" Content-Transfer-Encoding: quoted-printable

James,

 

I agree that privacy (of the end user = who has the MA) is important - in the ISP use case as well as the regulator= one

 

http= ://tools.ietf.org/html/draft-ietf-lmap-framework-01#section-8 discusses= privacy (data protection). It would be great to get your feedback on this = section. One idea that’s been raised is a Group-ID (section 8.7) R= 11; would be very interested to hear your opinion and also other ideas for = improving privacy (personally am not convinced by the Group-ID, as seems a = direct trade-off with security)

<= span style=3D'font-family:"Arial","sans-serif";color:blue'> 

thanks

phil<= /p>

 

 

From: James M= iller [mailto:jamesmilleresquire@gmail.com]
Sent: 06 November 20= 13 06:47
To: Nieminen Klaus
Cc: Eardley,PL,Philip,TUB8 = R; frode.sorensen@npt.no; lmap@ietf.org; Marc Linsner; Henning Schulzrinne<= br>Subject: Re: [lmap] LMAP regulator use case

=

 

I would propose adding in section 2 or 4 some text describing a r= egulators need to ensure that policies intended to protect the privacy inte= rests of a potential tester can be enforced using features available in the= system.  Recognizing that http://www.ietf.org/id/draft-folks-lmap-fram= ework-00.txt and in the lists there's a view that a single entity will = be overseeing a large scale measurement, there is a clear need for a test i= nfrastructure to include tools to enforce the privacy policies of the manag= ing entity and coordinate collaborative activity.  A desire to protect= the privacy of testers contributing to our mobile measurement effort lead = our work on the project, and our decision to adopt changes to our fixed tec= hnical infrastructure and data model to support a totally anonymous collect= ion at the MA and communications between controllers, and through the colle= ctors and the test results collected.  I understand there has been dis= cussion on this point in the past and we had included this in the Section 7= Security Considerations of our original I-D.  In any event I think th= e regulator use case benefits from calling the point out specifically.=

 

Propose= d addition in Section 2.2:

::=  Reg= ulators role in the development and enforcement of broadband Internet acces= s service policies also require that the measurement approaches meet a high= level of verifiability, accuracy and provider-independence to support vali= d and meaningful comparisons of Internet access service performance.  =  Regulators al= so require that policies intended to protect the privacy interests of poten= tial testers can be enforced using features available in the system.  =

 

=

…

= --_000_A2E337CDB7BC4145B018B9BEE8EB3E0D403B78C747EMV67UKRDdoma_-- From philip.eardley@bt.com Tue Nov 26 08:05:16 2013 Return-Path: X-Original-To: lmap@ietfa.amsl.com Delivered-To: lmap@ietfa.amsl.com Received: from localhost (ietfa.amsl.com [127.0.0.1]) by ietfa.amsl.com (Postfix) with ESMTP id 143AA1ADBCF for ; Tue, 26 Nov 2013 08:05:16 -0800 (PST) X-Virus-Scanned: amavisd-new at amsl.com X-Spam-Flag: NO X-Spam-Score: -1.935 X-Spam-Level: X-Spam-Status: No, score=-1.935 tagged_above=-999 required=5 tests=[BAYES_00=-1.9, RCVD_IN_DNSWL_LOW=-0.7, SPF_SOFTFAIL=0.665] autolearn=no Received: from mail.ietf.org ([4.31.198.44]) by localhost (ietfa.amsl.com [127.0.0.1]) (amavisd-new, port 10024) with ESMTP id 7SbZdm3rWqqc for ; Tue, 26 Nov 2013 08:05:14 -0800 (PST) Received: from smtpe1.intersmtp.com (smtp64.intersmtp.com [62.239.224.237]) by ietfa.amsl.com (Postfix) with ESMTP id CCD691ADEA6 for ; Tue, 26 Nov 2013 08:05:13 -0800 (PST) Received: from EVMHT63-UKRD.domain1.systemhost.net (10.36.3.100) by RDW083A008ED64.smtp-e4.hygiene.service (10.187.98.13) with Microsoft SMTP Server (TLS) id 8.3.298.1; Tue, 26 Nov 2013 16:05:12 +0000 Received: from EMV67-UKRD.domain1.systemhost.net ([169.254.1.152]) by EVMHT63-UKRD.domain1.systemhost.net ([10.36.3.100]) with mapi; Tue, 26 Nov 2013 16:05:12 +0000 From: To: , Date: Tue, 26 Nov 2013 16:05:12 +0000 Thread-Topic: [lmap] Comments on the current round of I-Ds, before the meeting tomorrow Thread-Index: Ac7aplg2ABTnDip0SbKDaRXBk9X57wQGZBpg Message-ID: References: <20131106041126.GA57287@idrathernotsay.com> In-Reply-To: <20131106041126.GA57287@idrathernotsay.com> Accept-Language: en-US, en-GB Content-Language: en-US X-MS-Has-Attach: X-MS-TNEF-Correlator: acceptlanguage: en-US, en-GB Content-Type: text/plain; charset="us-ascii" Content-Transfer-Encoding: quoted-printable MIME-Version: 1.0 Subject: Re: [lmap] Comments on the current round of I-Ds, before the meeting tomorrow X-BeenThere: lmap@ietf.org X-Mailman-Version: 2.1.15 Precedence: list List-Id: Large Scale Measurement of Access network Performance List-Unsubscribe: , List-Archive: List-Post: List-Help: List-Subscribe: , X-List-Received-Date: Tue, 26 Nov 2013 16:05:16 -0000 Steve, thanks for your careful reading - following up on the framework > LMAP Framework (draft-ietf-lmap-framework-01) > --------------------------------------------- > In general, the model here always has the MA connecting to the > collector to submit the results of a measurement. I've run into > situations before where it's been considered preferable to have the > collector connect outbound to the MAs to get the relevant results. Do > we have a way to address that? (Admittedly that's more difficult given > that many MAs are likely to be behind NATs; we'd have to address that > as well, even if we just pointed to suggestions as to how the MA could > keep a line of communication open. I know that's more or less a solved > problem even without resorting to uPNP or whatever, because I know that > some of the VOIP vendors do it, even though the solutions are all kinda > heedious.) [phil] ok, I'll try something like: the information flow is MA to Collector= , but it's possible the Collector initiates communications (similarly, the Control Protocol part is supposed to say that the informati= on flow is Controller to MA, but most likely the MA initiates communication= s as behind a NAT. ) >=20 > This may be a little controversial to say, but for a system of > significant scale it seems unlikely that using a SQL database for the > Results Database will work as well as one might hope (ask me how I > know). It might be worth calling that out, though that's a minor > tweak. [phil] I think the merits of sql, hadoop etc is a bit too detailed. >=20 > In 4.2, there's discussion about having just one controller at a time, > but I thought the HTTP draft said or implied that a MA might have more > than one controller telling it what to do. [phil] that's a comment for the http draft! >=20 > p. 14, at the top, discussion about what to do if the controller fails: > I'm not sure that's all that important to consider here. It seems like > the MA can just do another DNS lookup to locate the controller (and the > owner of the controller can change the IP address returned for the > controller if needed), or the owner of the controller can just stand up > another host at the same IP address -- in short, that seems like a > solved problem and in this case I'm not sure it's worth calling out the > solutions. [phil] agree, will delete this.=20 Thanks! phil From philip.eardley@bt.com Tue Nov 26 08:29:13 2013 Return-Path: X-Original-To: lmap@ietfa.amsl.com Delivered-To: lmap@ietfa.amsl.com Received: from localhost (ietfa.amsl.com [127.0.0.1]) by ietfa.amsl.com (Postfix) with ESMTP id 8D0931AD945 for ; Tue, 26 Nov 2013 08:29:13 -0800 (PST) X-Virus-Scanned: amavisd-new at amsl.com X-Spam-Flag: NO X-Spam-Score: -1.334 X-Spam-Level: X-Spam-Status: No, score=-1.334 tagged_above=-999 required=5 tests=[BAYES_00=-1.9, HTML_MESSAGE=0.001, J_CHICKENPOX_72=0.6, RCVD_IN_DNSWL_LOW=-0.7, SPF_SOFTFAIL=0.665] autolearn=no Received: from mail.ietf.org ([4.31.198.44]) by localhost (ietfa.amsl.com [127.0.0.1]) (amavisd-new, port 10024) with ESMTP id W0AIddveZe5q for ; Tue, 26 Nov 2013 08:29:10 -0800 (PST) Received: from smtpe1.intersmtp.com (smtp63.intersmtp.com [62.239.224.236]) by ietfa.amsl.com (Postfix) with ESMTP id 0920F1ACC91 for ; Tue, 26 Nov 2013 08:29:09 -0800 (PST) Received: from EVMHT67-UKRD.domain1.systemhost.net (10.36.3.104) by RDW083A007ED63.smtp-e3.hygiene.service (10.187.98.12) with Microsoft SMTP Server (TLS) id 8.3.298.1; Tue, 26 Nov 2013 16:29:08 +0000 Received: from EMV67-UKRD.domain1.systemhost.net ([169.254.1.152]) by EVMHT67-UKRD.domain1.systemhost.net ([10.36.3.104]) with mapi; Tue, 26 Nov 2013 16:29:08 +0000 From: To: , Date: Tue, 26 Nov 2013 16:29:08 +0000 Thread-Topic: 3 Suggested adds/ Edits for the LMAP Draft Thread-Index: Ac7bEh3r6TkvJxiSSsG+OjpDQgJCbAPr0UUw Message-ID: References: In-Reply-To: Accept-Language: en-US, en-GB Content-Language: en-US X-MS-Has-Attach: X-MS-TNEF-Correlator: acceptlanguage: en-US, en-GB Content-Type: multipart/alternative; boundary="_000_A2E337CDB7BC4145B018B9BEE8EB3E0D403B78C879EMV67UKRDdoma_" MIME-Version: 1.0 Subject: Re: [lmap] 3 Suggested adds/ Edits for the LMAP Draft X-BeenThere: lmap@ietf.org X-Mailman-Version: 2.1.15 Precedence: list List-Id: Large Scale Measurement of Access network Performance List-Unsubscribe: , List-Archive: List-Post: List-Help: List-Subscribe: , X-List-Received-Date: Tue, 26 Nov 2013 16:29:13 -0000 --_000_A2E337CDB7BC4145B018B9BEE8EB3E0D403B78C879EMV67UKRDdoma_ Content-Type: text/plain; charset="us-ascii" Content-Transfer-Encoding: quoted-printable Mike, Sorry for the slow response, been away a lot recently. Btw, the relevant draft is http://tools.ietf.org/html/draft-ietf-lmap-frame= work-01 (don't think there are too many differences, phew!) On your question - the idea is that the Measurement Peer could be a special= ised lmap device but could also be a general server, such as DNS or web ser= ver. On the terminology, would prefer not to adjust this unless we really need t= o. On the 2nd suggestion about Service Attributes, this is basically covered i= n the discussion of Subscriber Parameter Database (section 3, towards the e= nd of page 9). The reason I didn't try and craft a definition is that it's = in the 'out of scope' area of Figure 1 (so I figured that we wouldn't re-us= e the definition much) Thanks! Phil -- From: Bugenhagen, Michael K [mailto:Michael.K.Bugenhagen@centurylink.com] Sent: 06 November 2013 17:04 To: Eardley,PL,Philip,TUB8 R; 'lmap@ietf.org' Subject: 3 Suggested adds/ Edits for the LMAP Draft Phil, Reviewing the draft http://www.ietf.org/id/draft-folks-lmap-framework-00= .txt Add's (in Red) Page 6 definitions - Consider adding .... 1. Add Measurement Agent testing protocol Measurement Agent Protocol: The control messages from one M= A to another used to conduct a test. 2. Add Service Attributes Service Attributes: The normative Service attribute values= that are pertinent to a test results during the timeframe the test was con= ducted. Examples - Sold speed - may equal the current service capacity= attribute for the timeframe the test is conducted. In use - Service state - under maintenance, down, up, .... Tweaks - Page 11 LMAP phases - suggested change for the "measurement task being perf= ormed" * MA Test Engine The MA execution of actual Measurement Tasks are performed. A MA evaluates running tasks based on environment constraints, and = service attributes, and runs test tasks Recording (logs and captures test results) the type of test completion s= tate (successful, aborted, .....) It conducts Active Measurement Task involves sending test traffic betwee= n the Measurement Agent and a Measurement Peer, whilst a Passive Measurement Task involves (only) the Measurement A= gent observing existing user traffic. The LMAP WG does not define Measurement Methods, however the IPPM WG does. Questions - 1. Lack of a Consumer / subscriber test point Discover Method - The draft states that "comparable" test points are used to allow co= mparability between test results. Is the assumption that ONLY the controller presents the test points= , or we the framework leverage something broader and more transparent such = as a DNS or other registry for test points, logically some points ar= e already DNS named, but it seems we are adding some that do not normally g= et named, or we may what some ACL type restrictions to.... --_000_A2E337CDB7BC4145B018B9BEE8EB3E0D403B78C879EMV67UKRDdoma_ Content-Type: text/html; charset="us-ascii" Content-Transfer-Encoding: quoted-printable

Mike,

Sorry for the slow = response, been away a lot recently.

 

= Btw, the relevant draft is http://tools.ietf.org/= html/draft-ietf-lmap-framework-01

(don’t = think there are too many differences, phew!)

 

On your question – the idea is that the Measurement Peer co= uld be a specialised lmap device but could also be a general server, such a= s DNS or web server.

 

On the termin= ology, would prefer not to adjust this unless we really need to.

On the 2nd suggestion about Service Attributes, this is b= asically covered in the discussion of Subscriber Parameter Database (section 3, towa= rds the end of page 9). The reason I didn’t try and craft a definitio= n is that it’s in the ‘out of scope’ area of Figure 1 (so= I figured that we wouldn’t re-use the definition much)

Thanks!

Phil

 

--

 

From: = Bugenhagen, Michael K [mailto:Michael.K.Bugenhagen@centurylink.com]
= Sent: 06 November 2013 17:04
To: Eardley,PL,Philip,TUB8 R; 'l= map@ietf.org'
Subject: 3 Suggested adds/ Edits for the LMAP Draft=

 

Phil,

 <= /o:p>

   Reviewing the draft http://ww= w.ietf.org/id/draft-folks-lmap-framework-00.txt

 

Add= ’s (in Red)

 

Pag= e 6 definitions –  Consider adding ….

 

1.     Add Measurement A= gent testing protocol

&n= bsp;            = ;   Measurement Agent Protocol: The control messages from one MA= to another used to conduct a test.

 

2.  Add Service Attributes

         &nbs= p;      Service Attributes:  The normative Se= rvice attribute values that are pertinent to a test results during the time= frame the test was conducted.

           =      Examples –

         = ;            &n= bsp;  Sold speed - may equal the current service capacity attribute fo= r the timeframe the test is conducted.

         &nb= sp;            =   In use –

<= span style=3D'font-size:11.0pt;font-family:"Calibri","sans-serif";color:red= '>            &= nbsp;           Service s= tate – under maintenance, down, up, ….

 

Tweaks –

Page 11 L= MAP phases – suggested change for the “measurement task being p= erformed”

 =

·        = MA Test Engine

&n= bsp;       The MA e= xecution of actual Measurement Tasks are performed.  <= /span>

        A MA evaluates running tasks based on environment const= raints, and service attributes, and runs test tasks =

   Recording (logs and captures tes= t results) the type of test completion state (successful, aborted, ….= .)=

   It conducts Active Measurement Task involves sending test traffic between the Measurem= ent Agent and a Measurement Peer,

&nb= sp;       whilst a Passive Measurement Task i= nvolves (only) the Measurement Agent observing existing user traffic. = The

      LMAP= WG does not define Measurement Methods, however the IPPM WG

      does.

 

 =

Questions –

 

1.&= nbsp;      Lack of a Consu= mer / subscriber test point Discover Method –

        The draft state= s that “comparable” test points are used to allow comparability= between test results.

<= span style=3D'font-size:11.0pt;font-family:"Calibri","sans-serif"'> &n= bsp;      Is the assumption that ONLY the controll= er presents the test points, or we the framework leverage something broader= and more transparent such as a DNS or other registry for test  &= nbsp;     points, logically some points are already DNS= named, but it seems we are adding some that do not normally get named, or = we may what some ACL type restrictions to…. 

 

 

 

 

<= /body>= --_000_A2E337CDB7BC4145B018B9BEE8EB3E0D403B78C879EMV67UKRDdoma_-- From philip.eardley@bt.com Tue Nov 26 08:49:19 2013 Return-Path: X-Original-To: lmap@ietfa.amsl.com Delivered-To: lmap@ietfa.amsl.com Received: from localhost (ietfa.amsl.com [127.0.0.1]) by ietfa.amsl.com (Postfix) with ESMTP id 26C5A1A1F55 for ; Tue, 26 Nov 2013 08:49:19 -0800 (PST) X-Virus-Scanned: amavisd-new at amsl.com X-Spam-Flag: NO X-Spam-Score: -1.335 X-Spam-Level: X-Spam-Status: No, score=-1.335 tagged_above=-999 required=5 tests=[BAYES_00=-1.9, J_CHICKENPOX_72=0.6, RCVD_IN_DNSWL_LOW=-0.7, SPF_SOFTFAIL=0.665] autolearn=no Received: from mail.ietf.org ([4.31.198.44]) by localhost (ietfa.amsl.com [127.0.0.1]) (amavisd-new, port 10024) with ESMTP id 45U8lAW3x07d for ; Tue, 26 Nov 2013 08:49:16 -0800 (PST) Received: from smtpe1.intersmtp.com (smtp63.intersmtp.com [62.239.224.236]) by ietfa.amsl.com (Postfix) with ESMTP id A068F1AD8F0 for ; Tue, 26 Nov 2013 08:49:16 -0800 (PST) Received: from EVMHT62-UKRD.domain1.systemhost.net (10.36.3.128) by RDW083A007ED63.smtp-e3.hygiene.service (10.187.98.12) with Microsoft SMTP Server (TLS) id 8.3.298.1; Tue, 26 Nov 2013 16:49:15 +0000 Received: from EMV67-UKRD.domain1.systemhost.net ([169.254.1.152]) by EVMHT62-UKRD.domain1.systemhost.net ([10.36.3.128]) with mapi; Tue, 26 Nov 2013 16:49:15 +0000 From: To: Date: Tue, 26 Nov 2013 16:49:14 +0000 Thread-Topic: [lmap] Measurement Suppression broadcasting and non-acknowledgement Thread-Index: Ac7hG6uxiT9ROSAYTCyBe3e1T+uiPQJq4vSw Message-ID: References: <2D09D61DDFA73D4C884805CC7865E61130399373@GAALPA1MSGUSR9L.ITServices.sbc.com> <20131112163500.GD3992@elstar.local> <20131114092625.GA15158@elstar.local> In-Reply-To: <20131114092625.GA15158@elstar.local> Accept-Language: en-US, en-GB Content-Language: en-US X-MS-Has-Attach: X-MS-TNEF-Correlator: acceptlanguage: en-US, en-GB Content-Type: text/plain; charset="us-ascii" Content-Transfer-Encoding: quoted-printable MIME-Version: 1.0 Cc: lmap@ietf.org Subject: Re: [lmap] Measurement Suppression broadcasting and non-acknowledgement X-BeenThere: lmap@ietf.org X-Mailman-Version: 2.1.15 Precedence: list List-Id: Large Scale Measurement of Access network Performance List-Unsubscribe: , List-Archive: List-Post: List-Help: List-Subscribe: , X-List-Received-Date: Tue, 26 Nov 2013 16:49:19 -0000 I think you convinced me that for lmap 1.0 it would be better to go with on= e option, and this shouldn't be multicast Will update (currently do a revision on the framework) Thanks phil > -----Original Message----- > From: Juergen Schoenwaelder [mailto:j.schoenwaelder@jacobs- > university.de] > Sent: 14 November 2013 09:26 > To: Eardley,PL,Philip,TUB8 R > Cc: lmap@ietf.org > Subject: Re: [lmap] Measurement Suppression broadcasting and non- > acknowledgement >=20 > Phil, >=20 > so far I assumed we communicate over TLS/TCP and with that a multicast > make no sense at all. If you want to include the option to include > multicasts, then you have to answer a number of non-trivial questions: >=20 > - What is the transport? Since we talk wide area, it needs to be > congestion aware. >=20 > - How is the security provided over the transport? How are group > keys managed? >=20 > - How to make multicasts reliable? How to deal with MA that are > temporarily down? >=20 > - Why support multicasts for suppressions but not other changes > that might affect many MAs? >=20 > There is probably more. My point here (and I think this is also > Barbara's point) is that doing multicasts for the sake of saving a > bunch of ACKs adds significant challenges and so far I have not heard > about a system that does this 'optimization' today. >=20 > I believe this multicast idea shoud be _removed_ and not be made > optional - at least for LMAP 1.0. >=20 > /js >=20 > On Wed, Nov 13, 2013 at 04:32:50PM +0000, philip.eardley@bt.com wrote: > > Barbara > > Sounds Sensible, will rewrite to give both possibilities Thanks Phil > > ________________________________________ > > From: lmap-bounces@ietf.org [lmap-bounces@ietf.org] On Behalf Of > > Juergen Schoenwaelder [j.schoenwaelder@jacobs-university.de] > > Sent: 12 November 2013 16:35 > > To: STARK, BARBARA H > > Cc: lmap@ietf.org > > Subject: Re: [lmap] Measurement Suppression broadcasting and > > non-acknowledgement > > > > Barbara, > > > > I share your concerns. > > > > /js > > > > On Tue, Nov 12, 2013 at 04:17:01PM +0000, STARK, BARBARA H wrote: > > > I have some concerns with the way "Measurement Suppression" is > architected in framework currently. > > > > > > framework says "the "suppress" message is not acknowledged, since > it is likely to be broadcast to several /many MAs at a time when the > measurement system wants to eliminate inessential traffic." > > > > > > 1. I'm not sure if this is intended to literally mean "broadcast" > (which is a very specific concept in IP that uses broadcast destination > IP addresses like 255.255.255.255 or more specific subnet addresses to > reach all hosts on a network with a single sent message), or > "multicast" (similar to broadcast but sent to a multicast address), or > just massive quantities of unicast messages sent "simultaneously". > > > Where MAs are internal to an enterprise or campus network (managed > by Controllers also internal to the network), I could see where literal > IP broadcast messages (or even multicast messages) might be a good > technique to use. For MAs managed by Controllers that are out on the > Internet somewhere (which I'd hope we all agree are definitely in > scope), I don't think it's practical. Therefore I disagree that it's > likely. > > > > > > 2. I believe it would be a bad idea for MAs managed by Controllers > out on the Internet to react to a suppression message that is > unauthenticated and unencrypted. Therefore, there exists a need for it > to be possible for the Controller and each MA to establish > authenticated and encrypted (e.g., TLS) sessions prior to the > Controller sending the suppression command. > > > > > > 3. The overhead of having the MA send an Ack message after setting > up the authenticated and encrypted session isn't so big that it should > be prohibited, if the Controller operator wants such an Ack to occur. > > > > > > Therefore, I disagree that the framework should mandate non- > acknowledgement of suppression messages. I have no problem with > unacknowledged suppression messages being allowed -- this can be useful > in enterprise/campus environments. But I also believe there is value > in supporting acknowledged suppression messages, and that this option > should also be allowed. > > > _______________________________________________ > > > lmap mailing list > > > lmap@ietf.org > > > https://www.ietf.org/mailman/listinfo/lmap > > > > -- > > Juergen Schoenwaelder Jacobs University Bremen gGmbH > > Phone: +49 421 200 3587 Campus Ring 1, 28759 Bremen, Germany > > Fax: +49 421 200 3103 > > _______________________________________________ > > lmap mailing list > > lmap@ietf.org > > https://www.ietf.org/mailman/listinfo/lmap >=20 > -- > Juergen Schoenwaelder Jacobs University Bremen gGmbH > Phone: +49 421 200 3587 Campus Ring 1, 28759 Bremen, Germany > Fax: +49 421 200 3103 From philip.eardley@bt.com Wed Nov 27 07:12:04 2013 Return-Path: X-Original-To: lmap@ietfa.amsl.com Delivered-To: lmap@ietfa.amsl.com Received: from localhost (ietfa.amsl.com [127.0.0.1]) by ietfa.amsl.com (Postfix) with ESMTP id 40B421A802B for ; Wed, 27 Nov 2013 07:12:04 -0800 (PST) X-Virus-Scanned: amavisd-new at amsl.com X-Spam-Flag: NO X-Spam-Score: -1.335 X-Spam-Level: X-Spam-Status: No, score=-1.335 tagged_above=-999 required=5 tests=[BAYES_00=-1.9, J_CHICKENPOX_91=0.6, RCVD_IN_DNSWL_LOW=-0.7, SPF_SOFTFAIL=0.665] autolearn=no Received: from mail.ietf.org ([4.31.198.44]) by localhost (ietfa.amsl.com [127.0.0.1]) (amavisd-new, port 10024) with ESMTP id MbGEJuB7nm5L for ; Wed, 27 Nov 2013 07:11:58 -0800 (PST) Received: from smtpe1.intersmtp.com (smtp64.intersmtp.com [62.239.224.237]) by ietfa.amsl.com (Postfix) with ESMTP id 7FDC21ADBC9 for ; Wed, 27 Nov 2013 07:11:58 -0800 (PST) Received: from EVMHT67-UKRD.domain1.systemhost.net (10.36.3.104) by RDW083A008ED64.smtp-e4.hygiene.service (10.187.98.13) with Microsoft SMTP Server (TLS) id 8.3.298.1; Wed, 27 Nov 2013 15:11:56 +0000 Received: from EMV67-UKRD.domain1.systemhost.net ([169.254.1.152]) by EVMHT67-UKRD.domain1.systemhost.net ([10.36.3.104]) with mapi; Wed, 27 Nov 2013 15:11:56 +0000 From: To: , Date: Wed, 27 Nov 2013 15:11:55 +0000 Thread-Topic: Defining Status and Logging (Capabilities and Failure) Thread-Index: Ac7rgub539Fz9i0rRm6mrGWH6XJnhA== Message-ID: Accept-Language: en-US, en-GB Content-Language: en-US X-MS-Has-Attach: X-MS-TNEF-Correlator: acceptlanguage: en-US, en-GB Content-Type: text/plain; charset="us-ascii" Content-Transfer-Encoding: quoted-printable MIME-Version: 1.0 Cc: steve@idrathernotsay.com, lmap@ietf.org Subject: [lmap] Defining Status and Logging (Capabilities and Failure) X-BeenThere: lmap@ietf.org X-Mailman-Version: 2.1.15 Precedence: list List-Id: Large Scale Measurement of Access network Performance List-Unsubscribe: , List-Archive: List-Post: List-Help: List-Subscribe: , X-List-Received-Date: Wed, 27 Nov 2013 15:12:04 -0000 I agree with Juergen that we should be more consistent between the framewor= k & information model drafts. Currently the framework has (from MA to Controller): * ACKs of various msgs * List of Measurement Methods (typically in response to a Capability reques= t) * Failure report=20 Looking at Info model i-d: * Status Information - includes List of Measurement Methods and other items= like the interface speed and type and the link and IP address of the devic= e with the MA (btw, I think the section could be clearer - hopefully I got = it correct) * Logging Information - notify successful configuration of eg a reporting c= hannel; notify of a failure: the MA cannot action the Instruction (eg Task = is missing a mandatory parameter); Task couldn't be executed (eg unexpected= ly MA has no spare CPU); Collector not responding The wording is a bit confusing, eg logging information is said to include s= tatus. Firstly, I don't think the logging information should notify successful con= figuration of eg a reporting channel. The framework currently says: >> There is no need for the MA to confirm to the Controller that it has understood and acted on the Instruction, since the Controller knows the capabilities of the MA. However, the Control Protocol must support robust error reporting by the MA, to provide the Controller with sufficiently detailed reasons for any failures. Secondly, I suggest adding two definitions: Capabilities Information:=20 The list of the Measurement Methods that the MA can perform, plus informati= on about the device hosting the MA (for example its interface type and spee= d and its IP address).=20 Failure Information: Information about the MA's failure to action or execute an Instruction, whe= ther concerning Measurement Tasks or Reporting.=20 Also the Control Protocol definition would need tweaking to: Control Protocol: The protocol delivering Instruction(s) from a Controller = to a Measurement Agent. It also delivers Failure Information and Capabiliti= es information from the Measurement Agent to the Controller. =09 I prefer the term 'capabilities information' to 'status information' - pers= onally I think 'status' suggests 'is the MA on or off?'. Typically this msg is sent in response to a request from the Controller, bu= t it might also be sent when the MA becomes capable of a new MA, the rate o= f the interface changes or the MA re-boots. I prefer the term 'failure information' to 'status' - with the elimination = of notifying successful configuration, it now only includes info about fail= ures. I'd also be ok with 'error'. This msg is (only) sent asynchronously by the MA.=20 I'll also expand the framework's discussion of failures - at the moment it = only mentions Measurement Tasks - Reporting can also fail. Nit: in the framework i-d, currently the figure has "Failure report: (reaso= n)" - which I think is a typo for "[reason]" - as the nomenclature is (opti= onal) and [potentially repeated] Comments? Thanks Phil > -----Original Message----- > From: lmap-bounces@ietf.org [mailto:lmap-bounces@ietf.org] On Behalf Of > Juergen Schoenwaelder > Sent: 12 November 2013 11:20 > To: Burbridge,T,Trevor,TUB8 R > Cc: steve@idrathernotsay.com; lmap@ietf.org > Subject: Re: [lmap] Splitting measurement tasks into measurement tasks > plus reporting tasks? >=20 > On Tue, Nov 12, 2013 at 11:12:53AM +0000, trevor.burbridge@bt.com > wrote: > > > > In the information model the report channel delivers information to a > specified URL. I don't see why that can't be the Controller. Or if we > want we can talk about the Controller having a Collector interface we > can do that as well. This is all terminology. > > > > As I said we need to tidy up some of the naming and definitions, but > technically I don't see any problems - we designed it to work this way. > > >=20 > For me, this is not at all clear. In fact, the latest info model also > has a status channel and a logging channel. It is great if we can ship > stuff anywhere - but I think we would be even better off if we have a > clear view who should be the receiver / consumer of status and logging > information. I think this is actually an architectural question to > answer. >=20 > /js >=20 > -- > Juergen Schoenwaelder Jacobs University Bremen gGmbH > Phone: +49 421 200 3587 Campus Ring 1, 28759 Bremen, Germany > Fax: +49 421 200 3103 > _______________________________________________ > lmap mailing list > lmap@ietf.org > https://www.ietf.org/mailman/listinfo/lmap From philip.eardley@bt.com Wed Nov 27 08:48:19 2013 Return-Path: X-Original-To: lmap@ietfa.amsl.com Delivered-To: lmap@ietfa.amsl.com Received: from localhost (ietfa.amsl.com [127.0.0.1]) by ietfa.amsl.com (Postfix) with ESMTP id 2E7311ADEB4 for ; Wed, 27 Nov 2013 08:48:19 -0800 (PST) X-Virus-Scanned: amavisd-new at amsl.com X-Spam-Flag: NO X-Spam-Score: -1.335 X-Spam-Level: X-Spam-Status: No, score=-1.335 tagged_above=-999 required=5 tests=[BAYES_00=-1.9, J_CHICKENPOX_91=0.6, RCVD_IN_DNSWL_LOW=-0.7, SPF_SOFTFAIL=0.665] autolearn=no Received: from mail.ietf.org ([4.31.198.44]) by localhost (ietfa.amsl.com [127.0.0.1]) (amavisd-new, port 10024) with ESMTP id LUznuS1tslWB for ; Wed, 27 Nov 2013 08:48:15 -0800 (PST) Received: from smtpe1.intersmtp.com (smtp63.intersmtp.com [62.239.224.236]) by ietfa.amsl.com (Postfix) with ESMTP id 7CD911ADBCF for ; Wed, 27 Nov 2013 08:48:15 -0800 (PST) Received: from EVMHT61-UKRD.domain1.systemhost.net (10.36.3.127) by RDW083A007ED63.smtp-e3.hygiene.service (10.187.98.12) with Microsoft SMTP Server (TLS) id 8.3.298.1; Wed, 27 Nov 2013 16:48:13 +0000 Received: from EMV67-UKRD.domain1.systemhost.net ([169.254.1.152]) by EVMHT61-UKRD.domain1.systemhost.net ([10.36.3.127]) with mapi; Wed, 27 Nov 2013 16:48:13 +0000 From: To: , Date: Wed, 27 Nov 2013 16:48:12 +0000 Thread-Topic: Only Report raw measurement results? Thread-Index: Ac7rifIFtxdWQhQrS+WnMjRZ/vkZrQ== Message-ID: Accept-Language: en-US, en-GB Content-Language: en-US X-MS-Has-Attach: X-MS-TNEF-Correlator: acceptlanguage: en-US, en-GB Content-Type: text/plain; charset="us-ascii" Content-Transfer-Encoding: quoted-printable MIME-Version: 1.0 Cc: steve@idrathernotsay.com, lmap@ietf.org Subject: [lmap] Only Report raw measurement results? X-BeenThere: lmap@ietf.org X-Mailman-Version: 2.1.15 Precedence: list List-Id: Large Scale Measurement of Access network Performance List-Unsubscribe: , List-Archive: List-Post: List-Help: List-Subscribe: , X-List-Received-Date: Wed, 27 Nov 2013 16:48:19 -0000 Following up on Juergen's suggestion, how about adding text on these lines: In the initial LMAP Information Model and Report Protocol, for simplicity w= e assume that all Measurement Results are reported as-is, but allow extensi= bility so that a measurement system (or perhaps a second phase of LMAP) cou= ld allow a MA to pre-process Measurement Results before it reports them. Po= tential examples of pre-processing by the MA are: * labelling, or perhaps not including, Measurement Results impacted by for = instance cross-traffic or the MP being busy. =20 * detailing the start and end of suppression. * filtering outlier Results=20 * calculating some statistic like average (beyond that defined by the Measu= rement Task itself) * sending a special Report ("alarm") if recent Results match some pattern (= for example, below a threshold) * not sending a Report if there are no Measurement Results Are any of these things important enough (&/or simple enough) that they sho= uld be included in LMAP1.0? (some of them could be crafted by careful definition of Measurement Tasks -= eg filtering, calculating statistics and perhaps alarms) The above text could either fit at the end of S5.4 Report Protocol, or perh= aps in a new section "Items beyond the scope of the *initial* LMAP Protocol= Model". This actually raises a general point: the Info Model (and Protocol) i-ds sh= ould add some indication about extensibility.=20 Comments? Thanks phil > -----Original Message----- > From: lmap-bounces@ietf.org [mailto:lmap-bounces@ietf.org] On Behalf Of > Juergen Schoenwaelder > Sent: 14 November 2013 12:35 > To: Burbridge,T,Trevor,TUB8 R > Cc: steve@idrathernotsay.com; lmap@ietf.org > Subject: Re: [lmap] Splitting measurement tasks into measurement tasks > plus reporting tasks? >=20 > Trevor, >=20 > concerning (b) and (c), I think we should not consider this as a > requirement for LMAP 1.0 but perhaps we should consider how such > filtering and processing steps could be added in a later version of > LMAP. In other words, we may want to think about how we can make the > information model extensible such that (b) and (c) could be added > nicely in a later phase. >=20 > /js >=20 > On Wed, Nov 13, 2013 at 08:53:41AM +0000, trevor.burbridge@bt.com > wrote: > > (d) we have in the information model. > > > > (a)-(c) are interesting questions. (a) could always be a configurable > parameter. (b) and (c) I've kept away from for reasons of complexity, > but they could be useful tools. My initial reaction is that we don't > need pre-processing as (i) the amount of data returned is not likely to > be huge in most scenarios; (ii) some of this can be done within the > test - for example a stream of ping packets can be a single measurement > and report avg, min, max etc. > > > > Trevor. > > > > >-----Original Message----- > > >From: lmap-bounces@ietf.org [mailto:lmap-bounces@ietf.org] On Behalf > > >Of Juergen Schoenwaelder > > >Sent: 12 November 2013 19:26 > > >To: Steve Miller > > >Cc: lmap@ietf.org > > >Subject: Re: [lmap] Splitting measurement tasks into measurement > > >tasks plus reporting tasks? > > > > > >I think you touch on several issues: > > > > > >a) Are reports send if nothing is pending to be reported? > > > > > > I think the answer should be no but this can be discussed. > > > > > >b) Should we support a notion of filters that can be applied, that > is > > > only result records passing a filter expression are actually > reported. > > > > > >c) In a previous email, someone suggested to have aggregation > > > functions that can be applied generically to test results > > > (e.g., calculation of avg, min, max, ...). > > > > > >d) Should we support a report timing that kind of says "report > > > immediately if a result record is available" > > > > > >These are all fair questions one can ask and it is going to be a > > >trade-off between simplicity and functional richness. Perhaps we > need > > >to look at what deployed systems really do so we do not get over > > >board building aggregation and filtering features into the model > that > > >can also easily be applied on the collector's end. > > > > > >/js > > > > > >On Tue, Nov 12, 2013 at 01:56:11PM -0500, Steve Miller wrote: > > >> I think that's close but I'm not quite sure it's right. I > think > > >> part of the gotcha > > >here is that the spec tends to express Measurement Tasks as things > > >that have a definite start and end time, while I'm thinking of > things > > >that are running continuously. > > >> > > >> That is, if we wanted to do a DNS query once a second, forever, > > >> the current > > >model would express that more as: > > >> > > >> Measurement Task "DNS Performance": Perform a DNS > > >loss/latency/jitter check once a second, for 300 seconds, reporting > > >avg/min/max loss, latency and jitter plus a histogram of the > latencies. > > >> > > >> Report Channel "Collector C": Report to collector at > > >0,5,10,15,20,25,30,35,40,45,50,55 minutes after the hour. > > >> > > >> Measurement Schedule: Run the Measurement Task "DNS > Performance" > > >at 0,5,10,15,20,25,30,35,40,45,50,55 minutes after the hour and > > >report using Report Channel "Collector C". > > >> > > >> but each of those reports has a very definite start and end time - > - > > >> there's no > > >way there to define a report via a rolling window. > > >> > > >> In the current framework, to get what I'm thinking about, I can > > >> *almost* do > > >it, but I'd have to do that separately. First, to get the > > >performance numbers every five minutes, we'd have what's above, more > or less: > > >> > > >> Measurement Task "DNS Performance": Perform a DNS > > >loss/latency/jitter check once a second, for 300 seconds, reporting > > >avg/min/max loss, latency and jitter plus a histogram of the > latencies. > > >> > > >> Report Channel "Collector C 300Secs": Report to collector at > > >0,5,10,15,20,25,30,35,40,45,50,55 minutes after the hour. > > >> > > >> Measurement Schedule: Run the Measurement Task "DNS > Performance" > > >at 0,5,10,15,20,25,30,35,40,45,50,55 minutes after the hour and > > >report using Report Channel "Collector C 300Secs". > > >> > > >> and then to get some sort of report immediately, it seems like > > >> about as close as > > >we'd get would be: > > >> > > >> Measurement Task "DNS Badness": Perform a DNS > loss/latency/jitter > > >check once a second, for 300 seconds, reporting avg/min/max loss, > > >latency and jitter immediately if the loss is worse than 1%, the > > >latency is more than 200ms, or the jitter is more than 200ms. > > >> > > >> Report Channel "Collector C 1Sec": Report to collector once a > second. > > >> > > >> Measurement Schedule: Run the Measurement Task "DNS Badness" > at > > >0,5,10,15,20,25,30,35,40,45,50,55 minutes after the hour and report > > >using Report Channel "Collector C 1Sec". > > >> > > >> which has a couple of issues: > > >> > > >> - now we're sending twice as much traffic to each destination > as > > >> we were > > >before (which in this example isn't such a big deal but it's not > hard > > >to imagine that a file-download test or an RTP test wouldn't be so > > >forgiving (-: ) > > >> > > >> - it's not truly a rolling window, so you either have to > clamp the > > >accelerated reporting until you have enough baseline to be useful > > >(that is, you don't want to report "OMG! 100% loss!" if just the > > >first packet is lost) or you have to put up with noisy reports > > >> > > >> - I think the collector gets a report every second, even if > that > > >> report is > > >empty > > >> > > >> Maybe a way to look at this would be to have the concepts of: > > >> > > >> - never-ending measurements, which run continuously > > >> > > >> and > > >> > > >> - async report channels, where you could express simple > conditions > > >> to > > >control whether or not a report occurs > > >> > > >> or maybe async report channels are really some sort of async > > >> measurement > > >schedule: > > >> > > >> Run the test named "DNS Badness" continuously; report using > the > > >channel "Collector C" whenever loss is more than 1% or the latency > is > > >more than 200ms or the jitter is more than 200ms over the last 300 > seconds. > > >> > > >> I could be convinced that I'm thinking too hard about this or > > >> that my > > >perception of the measurement model as viewing Measurement Tasks as > > >discrete things with discrete start times and durations is simply > incorrect. > > >> > > >> Did that make anything more clear? > > >> > > >> Also, to address some of what you and Trevor were discussing: I > > >> don't know > > >that I necessarily regard these async reports as logging or status. > > >I think they're closer to "normal" reports, they just happen > > >asynchronously. I could be convinced to treat this as logging or > status, too, if that'd be helpful somehow. > > >> > > >> -Steve > > >> > > >> On Mon, Nov 11, 2013 at 09:42:18PM +0100, Juergen Schoenwaelder > wrote: > > >> > I hear two suggestions here: > > >> > > > >> > a) Give reporting channels their own independent schedule (right > now > > >> > they are essentially tied to the scheduled task). > > >> > > > >> > b) Provide a means for post-processing raw data in order to > derive > > >> > certain statistics before the result is shipped via a report > > >> > channel. > > >> > > > >> > Is this a fair summary of the request? > > >> _______________________________________________ > > >> lmap mailing list > > >> lmap@ietf.org > > >> https://www.ietf.org/mailman/listinfo/lmap > > > > > >-- > > >Juergen Schoenwaelder Jacobs University Bremen gGmbH > > >Phone: +49 421 200 3587 Campus Ring 1, 28759 Bremen, Germany > > >Fax: +49 421 200 3103 > > >_______________________________________________ > > >lmap mailing list > > >lmap@ietf.org > > >https://www.ietf.org/mailman/listinfo/lmap >=20 > -- > Juergen Schoenwaelder Jacobs University Bremen gGmbH > Phone: +49 421 200 3587 Campus Ring 1, 28759 Bremen, Germany > Fax: +49 421 200 3103 > _______________________________________________ > lmap mailing list > lmap@ietf.org > https://www.ietf.org/mailman/listinfo/lmap From philip.eardley@bt.com Wed Nov 27 08:58:54 2013 Return-Path: X-Original-To: lmap@ietfa.amsl.com Delivered-To: lmap@ietfa.amsl.com Received: from localhost (ietfa.amsl.com [127.0.0.1]) by ietfa.amsl.com (Postfix) with ESMTP id 827381AD7C0 for ; Wed, 27 Nov 2013 08:58:54 -0800 (PST) X-Virus-Scanned: amavisd-new at amsl.com X-Spam-Flag: NO X-Spam-Score: -1.935 X-Spam-Level: X-Spam-Status: No, score=-1.935 tagged_above=-999 required=5 tests=[BAYES_00=-1.9, RCVD_IN_DNSWL_LOW=-0.7, SPF_SOFTFAIL=0.665] autolearn=no Received: from mail.ietf.org ([4.31.198.44]) by localhost (ietfa.amsl.com [127.0.0.1]) (amavisd-new, port 10024) with ESMTP id 6krRnF1otdEj for ; Wed, 27 Nov 2013 08:58:52 -0800 (PST) Received: from smtpe1.intersmtp.com (smtp63.intersmtp.com [62.239.224.236]) by ietfa.amsl.com (Postfix) with ESMTP id 253481ACC91 for ; Wed, 27 Nov 2013 08:58:51 -0800 (PST) Received: from EVMHT62-UKRD.domain1.systemhost.net (10.36.3.128) by RDW083A007ED63.smtp-e3.hygiene.service (10.187.98.12) with Microsoft SMTP Server (TLS) id 8.3.298.1; Wed, 27 Nov 2013 16:58:50 +0000 Received: from EMV67-UKRD.domain1.systemhost.net ([169.254.1.152]) by EVMHT62-UKRD.domain1.systemhost.net ([10.36.3.128]) with mapi; Wed, 27 Nov 2013 16:58:50 +0000 From: To: , Date: Wed, 27 Nov 2013 16:58:48 +0000 Thread-Topic: degrees of Measurement Suppression Thread-Index: Ac7fxy6a9B101VqJQi6oteMX5LjSDgLrxZUw Message-ID: References: <2D09D61DDFA73D4C884805CC7865E611303993CD@GAALPA1MSGUSR9L.ITServices.sbc.com> In-Reply-To: <2D09D61DDFA73D4C884805CC7865E611303993CD@GAALPA1MSGUSR9L.ITServices.sbc.com> Accept-Language: en-US, en-GB Content-Language: en-US X-MS-Has-Attach: X-MS-TNEF-Correlator: acceptlanguage: en-US, en-GB Content-Type: text/plain; charset="us-ascii" Content-Transfer-Encoding: quoted-printable MIME-Version: 1.0 Subject: Re: [lmap] degrees of Measurement Suppression X-BeenThere: lmap@ietf.org X-Mailman-Version: 2.1.15 Precedence: list List-Id: Large Scale Measurement of Access network Performance List-Unsubscribe: , List-Archive: List-Post: List-Help: List-Subscribe: , X-List-Received-Date: Wed, 27 Nov 2013 16:58:54 -0000 (sorry, been away - catching up on mail backlog) Interesting idea. I'm inclined to think that, in lmap 1.0, we should keep it as simple as pos= sible and just go for suppress /don't suppress - as this would be easier to= implement for the MA and Controller, and avoid failure issues (eg MA not u= nderstanding what "suppress type 3" means). After all, suppression is inten= ded to be a temporary thing in an unexpected situation.=20 The multi-levels sounds a bit odd to me - something like: a few of my tests= create lots of load and should only be run when the network is really quie= t. Anyway, if you really want, you could get something like this by sending= the suppression message to a subset of the MAs.=20 If we really want this, then why not go a step further and allow the Contro= ller to signal (to a MA) arbitrary profiles of {sets of Measurement Tasks} = and then the Measurement Schedule references one of these profiles.=20 I'm not sure if anyone already operating a large-scale measurement system h= as found it necessary to go for multiple levels of suppression? If they hav= e, then I could be persuaded. If not, I think we should probably leave this= for lmap2.0. I think I agree that there might be a device-specific protocol (eg TR-069) = that can also signal suppress. Don't think the framework needs to mention t= his.=20 On a related topic, I think I heard someone mention the idea of the Measure= ment Peer sending a similar "suppress" message. I don't think an explicit m= sg is needed - but there are some related ideas in S5.3 (of the framework) = "Starting and stopping Measurement Tasks" Thanks phil > -----Original Message----- > From: lmap-bounces@ietf.org [mailto:lmap-bounces@ietf.org] On Behalf Of > STARK, BARBARA H > Sent: 12 November 2013 16:50 > To: lmap@ietf.org > Subject: [lmap] degrees of Measurement Suppression >=20 > A number of providers have discussed a model of Measurement Suppression > that supports more granular degrees of suppression (other than just > suppress and don't suppress). Michael Bugenhagen presented the original > version of this model, and he said it would be ok if I brought it up to > IETF. >=20 > Following are highlights of the proposal (with some modifications I've > included as a result of additional discussion): >=20 > 1. Measurement Methods (or perhaps Tasks) have a configurable parameter > that indicates whether the Controller operator considers them to be > "critical for OAM", "not critical, but not resource intensive", and > "not critical and resource intensive". >=20 > 2. The Controller can specify degrees of Measurement Suppression, which > should include: halt all non-critical tasks immediately but allow OAM > tasks; halt resource-intensive tasks immediately but allow non- > resource-intensive tasks; finish resource intensive tasks but do not > start new resource-intensive tasks and allow all non-resource-intensive > tasks; allow all tasks. >=20 > 3. It may also be possible for the unspecified bootstrap mechanism to > instruct the MA to suppress measurements. Where multiple channels exist > for Measurement Suppression (e.g., Controller and bootstrap), the MA is > to use the most restrictive setting. >=20 > Barbara >=20 >=20 > _______________________________________________ > lmap mailing list > lmap@ietf.org > https://www.ietf.org/mailman/listinfo/lmap From philip.eardley@bt.com Wed Nov 27 09:00:44 2013 Return-Path: X-Original-To: lmap@ietfa.amsl.com Delivered-To: lmap@ietfa.amsl.com Received: from localhost (ietfa.amsl.com [127.0.0.1]) by ietfa.amsl.com (Postfix) with ESMTP id 8C84D1ADF28 for ; Wed, 27 Nov 2013 09:00:44 -0800 (PST) X-Virus-Scanned: amavisd-new at amsl.com X-Spam-Flag: NO X-Spam-Score: -1.335 X-Spam-Level: X-Spam-Status: No, score=-1.335 tagged_above=-999 required=5 tests=[BAYES_00=-1.9, J_CHICKENPOX_46=0.6, RCVD_IN_DNSWL_LOW=-0.7, SPF_SOFTFAIL=0.665] autolearn=no Received: from mail.ietf.org ([4.31.198.44]) by localhost (ietfa.amsl.com [127.0.0.1]) (amavisd-new, port 10024) with ESMTP id N0STzTIvjo43 for ; Wed, 27 Nov 2013 09:00:41 -0800 (PST) Received: from smtpe1.intersmtp.com (smtp64.intersmtp.com [62.239.224.237]) by ietfa.amsl.com (Postfix) with ESMTP id 2D9321AD6B9 for ; Wed, 27 Nov 2013 09:00:41 -0800 (PST) Received: from EVMHT68-UKRD.domain1.systemhost.net (10.36.3.105) by RDW083A008ED64.smtp-e4.hygiene.service (10.187.98.13) with Microsoft SMTP Server (TLS) id 8.3.298.1; Wed, 27 Nov 2013 17:00:39 +0000 Received: from EMV67-UKRD.domain1.systemhost.net ([169.254.1.152]) by EVMHT68-UKRD.domain1.systemhost.net ([10.36.3.105]) with mapi; Wed, 27 Nov 2013 17:00:39 +0000 From: To: , Date: Wed, 27 Nov 2013 17:00:39 +0000 Thread-Topic: [lmap] Splitting measurement tasks into measurement tasks plus reporting tasks? Thread-Index: Ac7f2OiEg9cWYs41R1u8B7bwWsr7mQLse6IQ Message-ID: References: <20131111182839.GA21454@idrathernotsay.com> <20131111204218.GB770@elstar.local> <20131112185611.GA33370@idrathernotsay.com> In-Reply-To: <20131112185611.GA33370@idrathernotsay.com> Accept-Language: en-US, en-GB Content-Language: en-US X-MS-Has-Attach: X-MS-TNEF-Correlator: acceptlanguage: en-US, en-GB Content-Type: text/plain; charset="us-ascii" Content-Transfer-Encoding: quoted-printable MIME-Version: 1.0 Subject: Re: [lmap] Splitting measurement tasks into measurement tasks plus reporting tasks? X-BeenThere: lmap@ietf.org X-Mailman-Version: 2.1.15 Precedence: list List-Id: Large Scale Measurement of Access network Performance List-Unsubscribe: , List-Archive: List-Post: List-Help: List-Subscribe: , X-List-Received-Date: Wed, 27 Nov 2013 17:00:44 -0000 I take two questions from this discussion: 1. can a Measurement Task run indefinitely? 2. can a single Measurement Task create more than one Measurement Report? (= for instance, one the long term average, the other an immediate alarm that = a parameter has gone below a threshold, say) 1. I think this just needs some explanatory text on these lines: The Controller may want a MA to run the same Measurement Task indefinitely = (for example, "run the 'upload speed' Measurement Task once an hour until f= urther notice"). To avoid the MA generating traffic forever after a Control= ler has permanently failed, it is suggested that the Measurement Schedule i= ncludes a time limit ("run the 'upload speed' Measurement Task once an hour= for the next 30 days") and that the Measurement Schedule is updated regula= rly (say, every 10 days). Does this work? 2. think this needs a bit more discussion. I can see it might be useful (if= we have a strict 1:1 Task:Result, then the example above would need 2 Task= s and hence 2 lots of data to be sent. For some Tasks this would be ok, for= others not). So I'm OK if people think it won't add too much complexity to= the Information Model and Protocol.=20 (Incidentally this is in tension with the email "Only Report raw measuremen= t results". Perhaps we could instead describe this as two linked Tasks (eac= h with one Result) which in fact create only one pkt stream; perhaps this i= s just confusing.)=20 Thoughts? Thanks phil > -----Original Message----- > From: lmap-bounces@ietf.org [mailto:lmap-bounces@ietf.org] On Behalf Of > Steve Miller > Sent: 12 November 2013 18:56 > To: lmap@ietf.org > Subject: Re: [lmap] Splitting measurement tasks into measurement tasks > plus reporting tasks? >=20 > I think that's close but I'm not quite sure it's right. I think > part of the gotcha here is that the spec tends to express Measurement > Tasks as things that have a definite start and end time, while I'm > thinking of things that are running continuously. >=20 > That is, if we wanted to do a DNS query once a second, forever, the > current model would express that more as: >=20 > Measurement Task "DNS Performance": Perform a DNS > loss/latency/jitter check once a second, for 300 seconds, reporting > avg/min/max loss, latency and jitter plus a histogram of the latencies. >=20 > Report Channel "Collector C": Report to collector at > 0,5,10,15,20,25,30,35,40,45,50,55 minutes after the hour. >=20 > Measurement Schedule: Run the Measurement Task "DNS Performance" > at 0,5,10,15,20,25,30,35,40,45,50,55 minutes after the hour and report > using Report Channel "Collector C". >=20 > but each of those reports has a very definite start and end time -- > there's no way there to define a report via a rolling window. >=20 > In the current framework, to get what I'm thinking about, I can > *almost* do it, but I'd have to do that separately. First, to get the > performance numbers every five minutes, we'd have what's above, more or > less: >=20 > Measurement Task "DNS Performance": Perform a DNS > loss/latency/jitter check once a second, for 300 seconds, reporting > avg/min/max loss, latency and jitter plus a histogram of the latencies. >=20 > Report Channel "Collector C 300Secs": Report to collector at > 0,5,10,15,20,25,30,35,40,45,50,55 minutes after the hour. >=20 > Measurement Schedule: Run the Measurement Task "DNS Performance" > at 0,5,10,15,20,25,30,35,40,45,50,55 minutes after the hour and report > using Report Channel "Collector C 300Secs". >=20 > and then to get some sort of report immediately, it seems like about as > close as we'd get would be: >=20 > Measurement Task "DNS Badness": Perform a DNS loss/latency/jitter > check once a second, for 300 seconds, reporting avg/min/max loss, > latency and jitter immediately if the loss is worse than 1%, the > latency is more than 200ms, or the jitter is more than 200ms. >=20 > Report Channel "Collector C 1Sec": Report to collector once a > second. >=20 > Measurement Schedule: Run the Measurement Task "DNS Badness" at > 0,5,10,15,20,25,30,35,40,45,50,55 minutes after the hour and report > using Report Channel "Collector C 1Sec". >=20 > which has a couple of issues: >=20 > - now we're sending twice as much traffic to each destination as > we were before (which in this example isn't such a big deal but it's > not hard to imagine that a file-download test or an RTP test wouldn't > be so forgiving (-: ) >=20 > - it's not truly a rolling window, so you either have to clamp the > accelerated reporting until you have enough baseline to be useful (that > is, you don't want to report "OMG! 100% loss!" if just the first > packet is lost) or you have to put up with noisy reports >=20 > - I think the collector gets a report every second, even if that > report is empty >=20 > Maybe a way to look at this would be to have the concepts of: >=20 > - never-ending measurements, which run continuously >=20 > and >=20 > - async report channels, where you could express simple conditions > to control whether or not a report occurs >=20 > or maybe async report channels are really some sort of async > measurement schedule: >=20 > Run the test named "DNS Badness" continuously; report using the > channel "Collector C" whenever loss is more than 1% or the latency is > more than 200ms or the jitter is more than 200ms over the last 300 > seconds. >=20 > I could be convinced that I'm thinking too hard about this or that > my perception of the measurement model as viewing Measurement Tasks as > discrete things with discrete start times and durations is simply > incorrect. >=20 > Did that make anything more clear? >=20 > Also, to address some of what you and Trevor were discussing: I > don't know that I necessarily regard these async reports as logging or > status. I think they're closer to "normal" reports, they just happen > asynchronously. I could be convinced to treat this as logging or > status, too, if that'd be helpful somehow. >=20 > -Steve >=20 > On Mon, Nov 11, 2013 at 09:42:18PM +0100, Juergen Schoenwaelder wrote: > > I hear two suggestions here: > > > > a) Give reporting channels their own independent schedule (right now > > they are essentially tied to the scheduled task). > > > > b) Provide a means for post-processing raw data in order to derive > > certain statistics before the result is shipped via a report > > channel. > > > > Is this a fair summary of the request? > _______________________________________________ > lmap mailing list > lmap@ietf.org > https://www.ietf.org/mailman/listinfo/lmap From j.schoenwaelder@jacobs-university.de Wed Nov 27 23:00:55 2013 Return-Path: X-Original-To: lmap@ietfa.amsl.com Delivered-To: lmap@ietfa.amsl.com Received: from localhost (ietfa.amsl.com [127.0.0.1]) by ietfa.amsl.com (Postfix) with ESMTP id B9C221AE14E for ; Wed, 27 Nov 2013 23:00:55 -0800 (PST) X-Virus-Scanned: amavisd-new at amsl.com X-Spam-Flag: NO X-Spam-Score: -1.651 X-Spam-Level: X-Spam-Status: No, score=-1.651 tagged_above=-999 required=5 tests=[BAYES_00=-1.9, HELO_EQ_DE=0.35, J_CHICKENPOX_91=0.6, RCVD_IN_DNSWL_LOW=-0.7, RP_MATCHES_RCVD=-0.001] autolearn=no Received: from mail.ietf.org ([4.31.198.44]) by localhost (ietfa.amsl.com [127.0.0.1]) (amavisd-new, port 10024) with ESMTP id HSeH9_2BuNgF for ; Wed, 27 Nov 2013 23:00:53 -0800 (PST) Received: from hermes.jacobs-university.de (hermes.jacobs-university.de [212.201.44.23]) by ietfa.amsl.com (Postfix) with ESMTP id CC4751AE140 for ; Wed, 27 Nov 2013 23:00:51 -0800 (PST) Received: from localhost (demetrius2.jacobs-university.de [212.201.44.47]) by hermes.jacobs-university.de (Postfix) with ESMTP id 527BF20049; Thu, 28 Nov 2013 08:00:49 +0100 (CET) X-Virus-Scanned: amavisd-new at jacobs-university.de Received: from hermes.jacobs-university.de ([212.201.44.23]) by localhost (demetrius2.jacobs-university.de [212.201.44.32]) (amavisd-new, port 10024) with ESMTP id 3LCl1LDtHKIC; Thu, 28 Nov 2013 08:00:49 +0100 (CET) Received: from elstar.local (elstar.jacobs.jacobs-university.de [10.50.231.133]) by hermes.jacobs-university.de (Postfix) with ESMTP id B56BB2004A; Thu, 28 Nov 2013 08:00:46 +0100 (CET) Received: by elstar.local (Postfix, from userid 501) id 9FC532988E4B; Thu, 28 Nov 2013 08:00:41 +0100 (CET) Date: Thu, 28 Nov 2013 08:00:41 +0100 From: Juergen Schoenwaelder To: philip.eardley@bt.com Message-ID: <20131128070040.GA68751@elstar.local> Mail-Followup-To: philip.eardley@bt.com, trevor.burbridge@bt.com, steve@idrathernotsay.com, lmap@ietf.org References: MIME-Version: 1.0 Content-Type: text/plain; charset=us-ascii Content-Disposition: inline In-Reply-To: User-Agent: Mutt/1.5.21 (2010-09-15) Cc: trevor.burbridge@bt.com, steve@idrathernotsay.com, lmap@ietf.org Subject: Re: [lmap] Only Report raw measurement results? X-BeenThere: lmap@ietf.org X-Mailman-Version: 2.1.15 Precedence: list Reply-To: Juergen Schoenwaelder List-Id: Large Scale Measurement of Access network Performance List-Unsubscribe: , List-Archive: List-Post: List-Help: List-Subscribe: , X-List-Received-Date: Thu, 28 Nov 2013 07:00:56 -0000 Hi, I prefer to leave these things out of LMAP 1.0 but to think about how such raw result processing steps can be fitted into LMAP at a later point in time. There are several deployed systems that do not do this and hence it seems such raw data processing functions do not seem to be essential to get LMAP 1.0 out of the door. /js On Wed, Nov 27, 2013 at 04:48:12PM +0000, philip.eardley@bt.com wrote: > Following up on Juergen's suggestion, how about adding text on these lines: > > In the initial LMAP Information Model and Report Protocol, for simplicity we assume that all Measurement Results are reported as-is, but allow extensibility so that a measurement system (or perhaps a second phase of LMAP) could allow a MA to pre-process Measurement Results before it reports them. Potential examples of pre-processing by the MA are: > * labelling, or perhaps not including, Measurement Results impacted by for instance cross-traffic or the MP being busy. > * detailing the start and end of suppression. > * filtering outlier Results > * calculating some statistic like average (beyond that defined by the Measurement Task itself) > * sending a special Report ("alarm") if recent Results match some pattern (for example, below a threshold) > * not sending a Report if there are no Measurement Results > > Are any of these things important enough (&/or simple enough) that they should be included in LMAP1.0? > (some of them could be crafted by careful definition of Measurement Tasks - eg filtering, calculating statistics and perhaps alarms) > > > The above text could either fit at the end of S5.4 Report Protocol, or perhaps in a new section "Items beyond the scope of the *initial* LMAP Protocol Model". > > This actually raises a general point: the Info Model (and Protocol) i-ds should add some indication about extensibility. > > Comments? > Thanks > phil > > > -----Original Message----- > > From: lmap-bounces@ietf.org [mailto:lmap-bounces@ietf.org] On Behalf Of > > Juergen Schoenwaelder > > Sent: 14 November 2013 12:35 > > To: Burbridge,T,Trevor,TUB8 R > > Cc: steve@idrathernotsay.com; lmap@ietf.org > > Subject: Re: [lmap] Splitting measurement tasks into measurement tasks > > plus reporting tasks? > > > > Trevor, > > > > concerning (b) and (c), I think we should not consider this as a > > requirement for LMAP 1.0 but perhaps we should consider how such > > filtering and processing steps could be added in a later version of > > LMAP. In other words, we may want to think about how we can make the > > information model extensible such that (b) and (c) could be added > > nicely in a later phase. > > > > /js > > > > On Wed, Nov 13, 2013 at 08:53:41AM +0000, trevor.burbridge@bt.com > > wrote: > > > (d) we have in the information model. > > > > > > (a)-(c) are interesting questions. (a) could always be a configurable > > parameter. (b) and (c) I've kept away from for reasons of complexity, > > but they could be useful tools. My initial reaction is that we don't > > need pre-processing as (i) the amount of data returned is not likely to > > be huge in most scenarios; (ii) some of this can be done within the > > test - for example a stream of ping packets can be a single measurement > > and report avg, min, max etc. > > > > > > Trevor. > > > > > > >-----Original Message----- > > > >From: lmap-bounces@ietf.org [mailto:lmap-bounces@ietf.org] On Behalf > > > >Of Juergen Schoenwaelder > > > >Sent: 12 November 2013 19:26 > > > >To: Steve Miller > > > >Cc: lmap@ietf.org > > > >Subject: Re: [lmap] Splitting measurement tasks into measurement > > > >tasks plus reporting tasks? > > > > > > > >I think you touch on several issues: > > > > > > > >a) Are reports send if nothing is pending to be reported? > > > > > > > > I think the answer should be no but this can be discussed. > > > > > > > >b) Should we support a notion of filters that can be applied, that > > is > > > > only result records passing a filter expression are actually > > reported. > > > > > > > >c) In a previous email, someone suggested to have aggregation > > > > functions that can be applied generically to test results > > > > (e.g., calculation of avg, min, max, ...). > > > > > > > >d) Should we support a report timing that kind of says "report > > > > immediately if a result record is available" > > > > > > > >These are all fair questions one can ask and it is going to be a > > > >trade-off between simplicity and functional richness. Perhaps we > > need > > > >to look at what deployed systems really do so we do not get over > > > >board building aggregation and filtering features into the model > > that > > > >can also easily be applied on the collector's end. > > > > > > > >/js > > > > > > > >On Tue, Nov 12, 2013 at 01:56:11PM -0500, Steve Miller wrote: > > > >> I think that's close but I'm not quite sure it's right. I > > think > > > >> part of the gotcha > > > >here is that the spec tends to express Measurement Tasks as things > > > >that have a definite start and end time, while I'm thinking of > > things > > > >that are running continuously. > > > >> > > > >> That is, if we wanted to do a DNS query once a second, forever, > > > >> the current > > > >model would express that more as: > > > >> > > > >> Measurement Task "DNS Performance": Perform a DNS > > > >loss/latency/jitter check once a second, for 300 seconds, reporting > > > >avg/min/max loss, latency and jitter plus a histogram of the > > latencies. > > > >> > > > >> Report Channel "Collector C": Report to collector at > > > >0,5,10,15,20,25,30,35,40,45,50,55 minutes after the hour. > > > >> > > > >> Measurement Schedule: Run the Measurement Task "DNS > > Performance" > > > >at 0,5,10,15,20,25,30,35,40,45,50,55 minutes after the hour and > > > >report using Report Channel "Collector C". > > > >> > > > >> but each of those reports has a very definite start and end time - > > - > > > >> there's no > > > >way there to define a report via a rolling window. > > > >> > > > >> In the current framework, to get what I'm thinking about, I can > > > >> *almost* do > > > >it, but I'd have to do that separately. First, to get the > > > >performance numbers every five minutes, we'd have what's above, more > > or less: > > > >> > > > >> Measurement Task "DNS Performance": Perform a DNS > > > >loss/latency/jitter check once a second, for 300 seconds, reporting > > > >avg/min/max loss, latency and jitter plus a histogram of the > > latencies. > > > >> > > > >> Report Channel "Collector C 300Secs": Report to collector at > > > >0,5,10,15,20,25,30,35,40,45,50,55 minutes after the hour. > > > >> > > > >> Measurement Schedule: Run the Measurement Task "DNS > > Performance" > > > >at 0,5,10,15,20,25,30,35,40,45,50,55 minutes after the hour and > > > >report using Report Channel "Collector C 300Secs". > > > >> > > > >> and then to get some sort of report immediately, it seems like > > > >> about as close as > > > >we'd get would be: > > > >> > > > >> Measurement Task "DNS Badness": Perform a DNS > > loss/latency/jitter > > > >check once a second, for 300 seconds, reporting avg/min/max loss, > > > >latency and jitter immediately if the loss is worse than 1%, the > > > >latency is more than 200ms, or the jitter is more than 200ms. > > > >> > > > >> Report Channel "Collector C 1Sec": Report to collector once a > > second. > > > >> > > > >> Measurement Schedule: Run the Measurement Task "DNS Badness" > > at > > > >0,5,10,15,20,25,30,35,40,45,50,55 minutes after the hour and report > > > >using Report Channel "Collector C 1Sec". > > > >> > > > >> which has a couple of issues: > > > >> > > > >> - now we're sending twice as much traffic to each destination > > as > > > >> we were > > > >before (which in this example isn't such a big deal but it's not > > hard > > > >to imagine that a file-download test or an RTP test wouldn't be so > > > >forgiving (-: ) > > > >> > > > >> - it's not truly a rolling window, so you either have to > > clamp the > > > >accelerated reporting until you have enough baseline to be useful > > > >(that is, you don't want to report "OMG! 100% loss!" if just the > > > >first packet is lost) or you have to put up with noisy reports > > > >> > > > >> - I think the collector gets a report every second, even if > > that > > > >> report is > > > >empty > > > >> > > > >> Maybe a way to look at this would be to have the concepts of: > > > >> > > > >> - never-ending measurements, which run continuously > > > >> > > > >> and > > > >> > > > >> - async report channels, where you could express simple > > conditions > > > >> to > > > >control whether or not a report occurs > > > >> > > > >> or maybe async report channels are really some sort of async > > > >> measurement > > > >schedule: > > > >> > > > >> Run the test named "DNS Badness" continuously; report using > > the > > > >channel "Collector C" whenever loss is more than 1% or the latency > > is > > > >more than 200ms or the jitter is more than 200ms over the last 300 > > seconds. > > > >> > > > >> I could be convinced that I'm thinking too hard about this or > > > >> that my > > > >perception of the measurement model as viewing Measurement Tasks as > > > >discrete things with discrete start times and durations is simply > > incorrect. > > > >> > > > >> Did that make anything more clear? > > > >> > > > >> Also, to address some of what you and Trevor were discussing: I > > > >> don't know > > > >that I necessarily regard these async reports as logging or status. > > > >I think they're closer to "normal" reports, they just happen > > > >asynchronously. I could be convinced to treat this as logging or > > status, too, if that'd be helpful somehow. > > > >> > > > >> -Steve > > > >> > > > >> On Mon, Nov 11, 2013 at 09:42:18PM +0100, Juergen Schoenwaelder > > wrote: > > > >> > I hear two suggestions here: > > > >> > > > > >> > a) Give reporting channels their own independent schedule (right > > now > > > >> > they are essentially tied to the scheduled task). > > > >> > > > > >> > b) Provide a means for post-processing raw data in order to > > derive > > > >> > certain statistics before the result is shipped via a report > > > >> > channel. > > > >> > > > > >> > Is this a fair summary of the request? > > > >> _______________________________________________ > > > >> lmap mailing list > > > >> lmap@ietf.org > > > >> https://www.ietf.org/mailman/listinfo/lmap > > > > > > > >-- > > > >Juergen Schoenwaelder Jacobs University Bremen gGmbH > > > >Phone: +49 421 200 3587 Campus Ring 1, 28759 Bremen, Germany > > > >Fax: +49 421 200 3103 > > > >_______________________________________________ > > > >lmap mailing list > > > >lmap@ietf.org > > > >https://www.ietf.org/mailman/listinfo/lmap > > > > -- > > Juergen Schoenwaelder Jacobs University Bremen gGmbH > > Phone: +49 421 200 3587 Campus Ring 1, 28759 Bremen, Germany > > Fax: +49 421 200 3103 > > _______________________________________________ > > lmap mailing list > > lmap@ietf.org > > https://www.ietf.org/mailman/listinfo/lmap -- Juergen Schoenwaelder Jacobs University Bremen gGmbH Phone: +49 421 200 3587 Campus Ring 1, 28759 Bremen, Germany Fax: +49 421 200 3103 From j.schoenwaelder@jacobs-university.de Wed Nov 27 23:09:20 2013 Return-Path: X-Original-To: lmap@ietfa.amsl.com Delivered-To: lmap@ietfa.amsl.com Received: from localhost (ietfa.amsl.com [127.0.0.1]) by ietfa.amsl.com (Postfix) with ESMTP id 0B3BF1AE140 for ; Wed, 27 Nov 2013 23:09:20 -0800 (PST) X-Virus-Scanned: amavisd-new at amsl.com X-Spam-Flag: NO X-Spam-Score: -1.651 X-Spam-Level: X-Spam-Status: No, score=-1.651 tagged_above=-999 required=5 tests=[BAYES_00=-1.9, HELO_EQ_DE=0.35, J_CHICKENPOX_46=0.6, RCVD_IN_DNSWL_LOW=-0.7, RP_MATCHES_RCVD=-0.001] autolearn=no Received: from mail.ietf.org ([4.31.198.44]) by localhost (ietfa.amsl.com [127.0.0.1]) (amavisd-new, port 10024) with ESMTP id 3fhu6hp1nhKL for ; Wed, 27 Nov 2013 23:09:18 -0800 (PST) Received: from hermes.jacobs-university.de (hermes.jacobs-university.de [212.201.44.23]) by ietfa.amsl.com (Postfix) with ESMTP id C45651AE08B for ; Wed, 27 Nov 2013 23:09:17 -0800 (PST) Received: from localhost (demetrius4.jacobs-university.de [212.201.44.49]) by hermes.jacobs-university.de (Postfix) with ESMTP id B694B20050; Thu, 28 Nov 2013 08:09:15 +0100 (CET) X-Virus-Scanned: amavisd-new at jacobs-university.de Received: from hermes.jacobs-university.de ([212.201.44.23]) by localhost (demetrius4.jacobs-university.de [212.201.44.32]) (amavisd-new, port 10024) with ESMTP id 1glFCgB9aDZV; Thu, 28 Nov 2013 08:09:15 +0100 (CET) Received: from elstar.local (elstar.jacobs.jacobs-university.de [10.50.231.133]) by hermes.jacobs-university.de (Postfix) with ESMTP id 1D3572003F; Thu, 28 Nov 2013 08:09:15 +0100 (CET) Received: by elstar.local (Postfix, from userid 501) id BD3352988EEE; Thu, 28 Nov 2013 08:09:10 +0100 (CET) Date: Thu, 28 Nov 2013 08:09:10 +0100 From: Juergen Schoenwaelder To: philip.eardley@bt.com Message-ID: <20131128070910.GB68751@elstar.local> Mail-Followup-To: philip.eardley@bt.com, steve@idrathernotsay.com, lmap@ietf.org References: <20131111182839.GA21454@idrathernotsay.com> <20131111204218.GB770@elstar.local> <20131112185611.GA33370@idrathernotsay.com> MIME-Version: 1.0 Content-Type: text/plain; charset=us-ascii Content-Disposition: inline In-Reply-To: User-Agent: Mutt/1.5.21 (2010-09-15) Cc: steve@idrathernotsay.com, lmap@ietf.org Subject: Re: [lmap] Splitting measurement tasks into measurement tasks plus reporting tasks? X-BeenThere: lmap@ietf.org X-Mailman-Version: 2.1.15 Precedence: list Reply-To: Juergen Schoenwaelder List-Id: Large Scale Measurement of Access network Performance List-Unsubscribe: , List-Archive: List-Post: List-Help: List-Subscribe: , X-List-Received-Date: Thu, 28 Nov 2013 07:09:20 -0000 Hi, I think instructions must be considered some sort of soft state in general. An MA that for whatever reason looses connectivity to the controller for a very long time or permanently should stop doing measurements at some point in time and not continue indefinitely. Concerning the second point, I do not really see a problem with allowing a measurement task to create multiple result records. I can see that this is useful and easy to implement. I do not see how not allowing this would simplify anything. /js On Wed, Nov 27, 2013 at 05:00:39PM +0000, philip.eardley@bt.com wrote: > I take two questions from this discussion: > > 1. can a Measurement Task run indefinitely? > 2. can a single Measurement Task create more than one Measurement Report? (for instance, one the long term average, the other an immediate alarm that a parameter has gone below a threshold, say) > > 1. I think this just needs some explanatory text on these lines: > The Controller may want a MA to run the same Measurement Task indefinitely (for example, "run the 'upload speed' Measurement Task once an hour until further notice"). To avoid the MA generating traffic forever after a Controller has permanently failed, it is suggested that the Measurement Schedule includes a time limit ("run the 'upload speed' Measurement Task once an hour for the next 30 days") and that the Measurement Schedule is updated regularly (say, every 10 days). > > Does this work? > > 2. think this needs a bit more discussion. I can see it might be useful (if we have a strict 1:1 Task:Result, then the example above would need 2 Tasks and hence 2 lots of data to be sent. For some Tasks this would be ok, for others not). So I'm OK if people think it won't add too much complexity to the Information Model and Protocol. > > (Incidentally this is in tension with the email "Only Report raw measurement results". Perhaps we could instead describe this as two linked Tasks (each with one Result) which in fact create only one pkt stream; perhaps this is just confusing.) > > Thoughts? > > Thanks > phil > > > -----Original Message----- > > From: lmap-bounces@ietf.org [mailto:lmap-bounces@ietf.org] On Behalf Of > > Steve Miller > > Sent: 12 November 2013 18:56 > > To: lmap@ietf.org > > Subject: Re: [lmap] Splitting measurement tasks into measurement tasks > > plus reporting tasks? > > > > I think that's close but I'm not quite sure it's right. I think > > part of the gotcha here is that the spec tends to express Measurement > > Tasks as things that have a definite start and end time, while I'm > > thinking of things that are running continuously. > > > > That is, if we wanted to do a DNS query once a second, forever, the > > current model would express that more as: > > > > Measurement Task "DNS Performance": Perform a DNS > > loss/latency/jitter check once a second, for 300 seconds, reporting > > avg/min/max loss, latency and jitter plus a histogram of the latencies. > > > > Report Channel "Collector C": Report to collector at > > 0,5,10,15,20,25,30,35,40,45,50,55 minutes after the hour. > > > > Measurement Schedule: Run the Measurement Task "DNS Performance" > > at 0,5,10,15,20,25,30,35,40,45,50,55 minutes after the hour and report > > using Report Channel "Collector C". > > > > but each of those reports has a very definite start and end time -- > > there's no way there to define a report via a rolling window. > > > > In the current framework, to get what I'm thinking about, I can > > *almost* do it, but I'd have to do that separately. First, to get the > > performance numbers every five minutes, we'd have what's above, more or > > less: > > > > Measurement Task "DNS Performance": Perform a DNS > > loss/latency/jitter check once a second, for 300 seconds, reporting > > avg/min/max loss, latency and jitter plus a histogram of the latencies. > > > > Report Channel "Collector C 300Secs": Report to collector at > > 0,5,10,15,20,25,30,35,40,45,50,55 minutes after the hour. > > > > Measurement Schedule: Run the Measurement Task "DNS Performance" > > at 0,5,10,15,20,25,30,35,40,45,50,55 minutes after the hour and report > > using Report Channel "Collector C 300Secs". > > > > and then to get some sort of report immediately, it seems like about as > > close as we'd get would be: > > > > Measurement Task "DNS Badness": Perform a DNS loss/latency/jitter > > check once a second, for 300 seconds, reporting avg/min/max loss, > > latency and jitter immediately if the loss is worse than 1%, the > > latency is more than 200ms, or the jitter is more than 200ms. > > > > Report Channel "Collector C 1Sec": Report to collector once a > > second. > > > > Measurement Schedule: Run the Measurement Task "DNS Badness" at > > 0,5,10,15,20,25,30,35,40,45,50,55 minutes after the hour and report > > using Report Channel "Collector C 1Sec". > > > > which has a couple of issues: > > > > - now we're sending twice as much traffic to each destination as > > we were before (which in this example isn't such a big deal but it's > > not hard to imagine that a file-download test or an RTP test wouldn't > > be so forgiving (-: ) > > > > - it's not truly a rolling window, so you either have to clamp the > > accelerated reporting until you have enough baseline to be useful (that > > is, you don't want to report "OMG! 100% loss!" if just the first > > packet is lost) or you have to put up with noisy reports > > > > - I think the collector gets a report every second, even if that > > report is empty > > > > Maybe a way to look at this would be to have the concepts of: > > > > - never-ending measurements, which run continuously > > > > and > > > > - async report channels, where you could express simple conditions > > to control whether or not a report occurs > > > > or maybe async report channels are really some sort of async > > measurement schedule: > > > > Run the test named "DNS Badness" continuously; report using the > > channel "Collector C" whenever loss is more than 1% or the latency is > > more than 200ms or the jitter is more than 200ms over the last 300 > > seconds. > > > > I could be convinced that I'm thinking too hard about this or that > > my perception of the measurement model as viewing Measurement Tasks as > > discrete things with discrete start times and durations is simply > > incorrect. > > > > Did that make anything more clear? > > > > Also, to address some of what you and Trevor were discussing: I > > don't know that I necessarily regard these async reports as logging or > > status. I think they're closer to "normal" reports, they just happen > > asynchronously. I could be convinced to treat this as logging or > > status, too, if that'd be helpful somehow. > > > > -Steve > > > > On Mon, Nov 11, 2013 at 09:42:18PM +0100, Juergen Schoenwaelder wrote: > > > I hear two suggestions here: > > > > > > a) Give reporting channels their own independent schedule (right now > > > they are essentially tied to the scheduled task). > > > > > > b) Provide a means for post-processing raw data in order to derive > > > certain statistics before the result is shipped via a report > > > channel. > > > > > > Is this a fair summary of the request? > > _______________________________________________ > > lmap mailing list > > lmap@ietf.org > > https://www.ietf.org/mailman/listinfo/lmap > _______________________________________________ > lmap mailing list > lmap@ietf.org > https://www.ietf.org/mailman/listinfo/lmap -- Juergen Schoenwaelder Jacobs University Bremen gGmbH Phone: +49 421 200 3587 Campus Ring 1, 28759 Bremen, Germany Fax: +49 421 200 3103 From j.schoenwaelder@jacobs-university.de Thu Nov 28 08:48:08 2013 Return-Path: X-Original-To: lmap@ietfa.amsl.com Delivered-To: lmap@ietfa.amsl.com Received: from localhost (ietfa.amsl.com [127.0.0.1]) by ietfa.amsl.com (Postfix) with ESMTP id C53CD1AD73F for ; Thu, 28 Nov 2013 08:48:08 -0800 (PST) X-Virus-Scanned: amavisd-new at amsl.com X-Spam-Flag: NO X-Spam-Score: -1.651 X-Spam-Level: X-Spam-Status: No, score=-1.651 tagged_above=-999 required=5 tests=[BAYES_00=-1.9, HELO_EQ_DE=0.35, J_CHICKENPOX_91=0.6, RCVD_IN_DNSWL_LOW=-0.7, RP_MATCHES_RCVD=-0.001] autolearn=no Received: from mail.ietf.org ([4.31.198.44]) by localhost (ietfa.amsl.com [127.0.0.1]) (amavisd-new, port 10024) with ESMTP id r7yCoXO7Tpne for ; Thu, 28 Nov 2013 08:48:06 -0800 (PST) Received: from hermes.jacobs-university.de (hermes.jacobs-university.de [212.201.44.23]) by ietfa.amsl.com (Postfix) with ESMTP id 2215F1AD73E for ; Thu, 28 Nov 2013 08:48:06 -0800 (PST) Received: from localhost (demetrius1.jacobs-university.de [212.201.44.46]) by hermes.jacobs-university.de (Postfix) with ESMTP id B36B220065; Thu, 28 Nov 2013 17:48:04 +0100 (CET) X-Virus-Scanned: amavisd-new at jacobs-university.de Received: from hermes.jacobs-university.de ([212.201.44.23]) by localhost (demetrius1.jacobs-university.de [212.201.44.32]) (amavisd-new, port 10024) with ESMTP id u-mnarbai-rI; Thu, 28 Nov 2013 17:48:04 +0100 (CET) Received: from elstar.local (elstar.jacobs.jacobs-university.de [10.50.231.133]) by hermes.jacobs-university.de (Postfix) with ESMTP id 1F23A20063; Thu, 28 Nov 2013 17:48:04 +0100 (CET) Received: by elstar.local (Postfix, from userid 501) id 96328298AA44; Thu, 28 Nov 2013 17:47:58 +0100 (CET) Date: Thu, 28 Nov 2013 17:47:57 +0100 From: Juergen Schoenwaelder To: philip.eardley@bt.com Message-ID: <20131128164756.GA70319@elstar.local> Mail-Followup-To: philip.eardley@bt.com, trevor.burbridge@bt.com, steve@idrathernotsay.com, lmap@ietf.org References: MIME-Version: 1.0 Content-Type: text/plain; charset=us-ascii Content-Disposition: inline In-Reply-To: User-Agent: Mutt/1.5.21 (2010-09-15) Cc: trevor.burbridge@bt.com, steve@idrathernotsay.com, lmap@ietf.org Subject: Re: [lmap] Defining Status and Logging (Capabilities and Failure) X-BeenThere: lmap@ietf.org X-Mailman-Version: 2.1.15 Precedence: list Reply-To: Juergen Schoenwaelder List-Id: Large Scale Measurement of Access network Performance List-Unsubscribe: , List-Archive: List-Post: List-Help: List-Subscribe: , X-List-Received-Date: Thu, 28 Nov 2013 16:48:09 -0000 Phil, I agree with many things you wrote but I am still unclear about 'status' information. For me, capabilities are relatively static, e.g., I do not expect that the list of measurement methods changes highly dynamically. Other information like say the IP addresses of an interface are what management people meanwhile call operational state and this can change regularly (e.g., because DHCP leases timeout). So lumping all this together into 'capabilities' may not be right. That said, I personally believe the status information is right now too detailed in the information model. For some of the relevant status information we do have existing data models and I think we should promote their reuse rather than defining new lists of objects in the LMAP information model. And I think we need to focus on the _essential_ information a _controller_ needs to generate proper instructions. For data collection, I can imagine to install a measurement test that reports say the local interface configuration and the IP layer network settings. This way, LMAP does not need a special mechanism to send operational state for data collection purposes. Perhaps we should change the category to "Capability and Status Information" or we split things such that "Capability Information" is really just capability information and "Status Information" is really just the _essential_ operational state need by the controller to generate Instructions. /js On Wed, Nov 27, 2013 at 03:11:55PM +0000, philip.eardley@bt.com wrote: > I agree with Juergen that we should be more consistent between the framework & information model drafts. > > Currently the framework has (from MA to Controller): > * ACKs of various msgs > * List of Measurement Methods (typically in response to a Capability request) > * Failure report > > Looking at Info model i-d: > * Status Information - includes List of Measurement Methods and other items like the interface speed and type and the link and IP address of the device with the MA (btw, I think the section could be clearer - hopefully I got it correct) > * Logging Information - notify successful configuration of eg a reporting channel; notify of a failure: the MA cannot action the Instruction (eg Task is missing a mandatory parameter); Task couldn't be executed (eg unexpectedly MA has no spare CPU); Collector not responding > > The wording is a bit confusing, eg logging information is said to include status. > > Firstly, I don't think the logging information should notify successful configuration of eg a reporting channel. The framework currently says: > >> There is no need for the MA to confirm to the Controller that it has > understood and acted on the Instruction, since the Controller knows > the capabilities of the MA. However, the Control Protocol must > support robust error reporting by the MA, to provide the Controller > with sufficiently detailed reasons for any failures. > > Secondly, I suggest adding two definitions: > > Capabilities Information: > The list of the Measurement Methods that the MA can perform, plus information about the device hosting the MA (for example its interface type and speed and its IP address). > > Failure Information: > Information about the MA's failure to action or execute an Instruction, whether concerning Measurement Tasks or Reporting. > > Also the Control Protocol definition would need tweaking to: > Control Protocol: The protocol delivering Instruction(s) from a Controller to a Measurement Agent. It also delivers Failure Information and Capabilities information from the Measurement Agent to the Controller. > > > I prefer the term 'capabilities information' to 'status information' - personally I think 'status' suggests 'is the MA on or off?'. > Typically this msg is sent in response to a request from the Controller, but it might also be sent when the MA becomes capable of a new MA, the rate of the interface changes or the MA re-boots. > > I prefer the term 'failure information' to 'status' - with the elimination of notifying successful configuration, it now only includes info about failures. I'd also be ok with 'error'. > This msg is (only) sent asynchronously by the MA. > > I'll also expand the framework's discussion of failures - at the moment it only mentions Measurement Tasks - Reporting can also fail. > > Nit: in the framework i-d, currently the figure has "Failure report: (reason)" - which I think is a typo for "[reason]" - as the nomenclature is (optional) and [potentially repeated] > > Comments? > Thanks > Phil > > > -----Original Message----- > > From: lmap-bounces@ietf.org [mailto:lmap-bounces@ietf.org] On Behalf Of > > Juergen Schoenwaelder > > Sent: 12 November 2013 11:20 > > To: Burbridge,T,Trevor,TUB8 R > > Cc: steve@idrathernotsay.com; lmap@ietf.org > > Subject: Re: [lmap] Splitting measurement tasks into measurement tasks > > plus reporting tasks? > > > > On Tue, Nov 12, 2013 at 11:12:53AM +0000, trevor.burbridge@bt.com > > wrote: > > > > > > In the information model the report channel delivers information to a > > specified URL. I don't see why that can't be the Controller. Or if we > > want we can talk about the Controller having a Collector interface we > > can do that as well. This is all terminology. > > > > > > As I said we need to tidy up some of the naming and definitions, but > > technically I don't see any problems - we designed it to work this way. > > > > > > > For me, this is not at all clear. In fact, the latest info model also > > has a status channel and a logging channel. It is great if we can ship > > stuff anywhere - but I think we would be even better off if we have a > > clear view who should be the receiver / consumer of status and logging > > information. I think this is actually an architectural question to > > answer. > > > > /js > > > > -- > > Juergen Schoenwaelder Jacobs University Bremen gGmbH > > Phone: +49 421 200 3587 Campus Ring 1, 28759 Bremen, Germany > > Fax: +49 421 200 3103 > > _______________________________________________ > > lmap mailing list > > lmap@ietf.org > > https://www.ietf.org/mailman/listinfo/lmap -- Juergen Schoenwaelder Jacobs University Bremen gGmbH Phone: +49 421 200 3587 Campus Ring 1, 28759 Bremen, Germany Fax: +49 421 200 3103 From philip.eardley@bt.com Fri Nov 29 02:56:41 2013 Return-Path: X-Original-To: lmap@ietfa.amsl.com Delivered-To: lmap@ietfa.amsl.com Received: from localhost (ietfa.amsl.com [127.0.0.1]) by ietfa.amsl.com (Postfix) with ESMTP id 269D11AD8F4 for ; Fri, 29 Nov 2013 02:56:41 -0800 (PST) X-Virus-Scanned: amavisd-new at amsl.com X-Spam-Flag: NO X-Spam-Score: -0.735 X-Spam-Level: X-Spam-Status: No, score=-0.735 tagged_above=-999 required=5 tests=[BAYES_00=-1.9, J_CHICKENPOX_72=0.6, J_CHICKENPOX_91=0.6, RCVD_IN_DNSWL_LOW=-0.7, SPF_SOFTFAIL=0.665] autolearn=no Received: from mail.ietf.org ([4.31.198.44]) by localhost (ietfa.amsl.com [127.0.0.1]) (amavisd-new, port 10024) with ESMTP id VPUaqs19VR_y for ; Fri, 29 Nov 2013 02:56:39 -0800 (PST) Received: from smtpe1.intersmtp.com (smtpe1.intersmtp.com [62.239.224.235]) by ietfa.amsl.com (Postfix) with ESMTP id 8B53D1ACCE5 for ; Fri, 29 Nov 2013 02:56:38 -0800 (PST) Received: from EVMHT62-UKRD.domain1.systemhost.net (10.36.3.128) by RDW083A006ED62.bt.com (10.187.98.11) with Microsoft SMTP Server (TLS) id 14.3.169.1; Fri, 29 Nov 2013 10:56:33 +0000 Received: from EMV67-UKRD.domain1.systemhost.net ([169.254.1.152]) by EVMHT62-UKRD.domain1.systemhost.net ([10.36.3.128]) with mapi; Fri, 29 Nov 2013 10:56:06 +0000 From: To: Date: Fri, 29 Nov 2013 10:56:05 +0000 Thread-Topic: Defining Status and Logging (Capabilities and Failure) Thread-Index: Ac7sWaEwWg1tM/A3SGyElTTnFvuZRQAkUuhA Message-ID: References: <20131128164756.GA70319@elstar.local> In-Reply-To: <20131128164756.GA70319@elstar.local> Accept-Language: en-US, en-GB Content-Language: en-US X-MS-Has-Attach: X-MS-TNEF-Correlator: acceptlanguage: en-US, en-GB Content-Type: text/plain; charset="us-ascii" Content-Transfer-Encoding: quoted-printable MIME-Version: 1.0 Cc: trevor.burbridge@bt.com, steve@idrathernotsay.com, lmap@ietf.org Subject: Re: [lmap] Defining Status and Logging (Capabilities and Failure) X-BeenThere: lmap@ietf.org X-Mailman-Version: 2.1.15 Precedence: list List-Id: Large Scale Measurement of Access network Performance List-Unsubscribe: , List-Archive: List-Post: List-Help: List-Subscribe: , X-List-Received-Date: Fri, 29 Nov 2013 10:56:41 -0000 Capabilities vs operational info seems a reasonable split to me. Yes, it's always possible to report info to the Collector and from there (v= ia an out-of-scope mechanism) to the Controller. The example we gave was th= e results of one test leading to some follow-up tests. Maybe this type of i= nteraction should be assumed where we have to analyse the results before de= ciding what Task to do next - if you're trying to diagnose a fault.=20 Whereas if you're trying to decide how big a file to use for your download = speed test, that depends on operational state (the interface speed) The info model has lots of optional stuff concerning operational state. Who= decides what's sent? Does the Controller's Request-for-operational-info in= clude the list of things it wants to learn? Or not - and then does the MA r= eply with all Operational-info? Everything that's changed since it last sen= t? Any operational-info at its choice? Do we allow extensibility? Ie can the operator of the measurement system in= clude its own additional operational-info fields? If yes, how? My guesses would be: Request-for-operational-info(list) or (all) Can define own operator /vendor specific fields I also think it's worth adding to the info model i-d reasons why the Inform= ation Model divided up at all. I guess it's something like: 1. different information transferred between different elements, eg the Ins= truction information is transmitted from the Controller to the MA, the Repo= rting from the MA to the Collector, pre-configuration information ... 2. between the same functions, but at different timescales and at different= times. Eg for the sub-parts of the Instruction, the Measurement Tasks & re= porting channels would be updated infrequently; the Measurement Schedule qu= ite often, and suppression ad hoc when needed... 3. possibly the protocol can be different /optimised for each case - or re-= use of a data model, as you mention 4. anything else? phil > -----Original Message----- > From: Juergen Schoenwaelder [mailto:j.schoenwaelder@jacobs- > university.de] > Sent: 28 November 2013 16:48 > To: Eardley,PL,Philip,TUB8 R > Cc: Burbridge,T,Trevor,TUB8 R; steve@idrathernotsay.com; lmap@ietf.org > Subject: Re: Defining Status and Logging (Capabilities and Failure) >=20 > Phil, >=20 > I agree with many things you wrote but I am still unclear about > 'status' information. For me, capabilities are relatively static, e.g., > I do not expect that the list of measurement methods changes highly > dynamically. Other information like say the IP addresses of an > interface are what management people meanwhile call operational state > and this can change regularly (e.g., because DHCP leases timeout). So > lumping all this together into 'capabilities' may not be right. >=20 > That said, I personally believe the status information is right now too > detailed in the information model. For some of the relevant status > information we do have existing data models and I think we should > promote their reuse rather than defining new lists of objects in the > LMAP information model. And I think we need to focus on the _essential_ > information a _controller_ needs to generate proper instructions. >=20 > For data collection, I can imagine to install a measurement test that > reports say the local interface configuration and the IP layer network > settings. This way, LMAP does not need a special mechanism to send > operational state for data collection purposes. >=20 >=20 > Perhaps we should change the category to "Capability and Status > Information" or we split things such that "Capability Information" is > really just capability information and "Status Information" is really > just the _essential_ operational state need by the controller to > generate Instructions. >=20 > /js >=20 > On Wed, Nov 27, 2013 at 03:11:55PM +0000, philip.eardley@bt.com wrote: > > I agree with Juergen that we should be more consistent between the > framework & information model drafts. > > > > Currently the framework has (from MA to Controller): > > * ACKs of various msgs > > * List of Measurement Methods (typically in response to a Capability > > request) > > * Failure report > > > > Looking at Info model i-d: > > * Status Information - includes List of Measurement Methods and other > > items like the interface speed and type and the link and IP address > of > > the device with the MA (btw, I think the section could be clearer - > > hopefully I got it correct) > > * Logging Information - notify successful configuration of eg a > > reporting channel; notify of a failure: the MA cannot action the > > Instruction (eg Task is missing a mandatory parameter); Task couldn't > > be executed (eg unexpectedly MA has no spare CPU); Collector not > > responding > > > > The wording is a bit confusing, eg logging information is said to > include status. > > > > Firstly, I don't think the logging information should notify > successful configuration of eg a reporting channel. The framework > currently says: > > >> There is no need for the MA to confirm to the Controller that it > > >> has > > understood and acted on the Instruction, since the Controller > knows > > the capabilities of the MA. However, the Control Protocol must > > support robust error reporting by the MA, to provide the > Controller > > with sufficiently detailed reasons for any failures. > > > > Secondly, I suggest adding two definitions: > > > > Capabilities Information: > > The list of the Measurement Methods that the MA can perform, plus > information about the device hosting the MA (for example its interface > type and speed and its IP address). > > > > Failure Information: > > Information about the MA's failure to action or execute an > Instruction, whether concerning Measurement Tasks or Reporting. > > > > Also the Control Protocol definition would need tweaking to: > > Control Protocol: The protocol delivering Instruction(s) from a > Controller to a Measurement Agent. It also delivers Failure Information > and Capabilities information from the Measurement Agent to the > Controller. > > > > > > I prefer the term 'capabilities information' to 'status information' > - personally I think 'status' suggests 'is the MA on or off?'. > > Typically this msg is sent in response to a request from the > Controller, but it might also be sent when the MA becomes capable of a > new MA, the rate of the interface changes or the MA re-boots. > > > > I prefer the term 'failure information' to 'status' - with the > elimination of notifying successful configuration, it now only includes > info about failures. I'd also be ok with 'error'. > > This msg is (only) sent asynchronously by the MA. > > > > I'll also expand the framework's discussion of failures - at the > moment it only mentions Measurement Tasks - Reporting can also fail. > > > > Nit: in the framework i-d, currently the figure has "Failure report: > > (reason)" - which I think is a typo for "[reason]" - as the > > nomenclature is (optional) and [potentially repeated] > > > > Comments? > > Thanks > > Phil > > > > > -----Original Message----- > > > From: lmap-bounces@ietf.org [mailto:lmap-bounces@ietf.org] On > Behalf > > > Of Juergen Schoenwaelder > > > Sent: 12 November 2013 11:20 > > > To: Burbridge,T,Trevor,TUB8 R > > > Cc: steve@idrathernotsay.com; lmap@ietf.org > > > Subject: Re: [lmap] Splitting measurement tasks into measurement > > > tasks plus reporting tasks? > > > > > > On Tue, Nov 12, 2013 at 11:12:53AM +0000, trevor.burbridge@bt.com > > > wrote: > > > > > > > > In the information model the report channel delivers information > > > > to a > > > specified URL. I don't see why that can't be the Controller. Or if > > > we want we can talk about the Controller having a Collector > > > interface we can do that as well. This is all terminology. > > > > > > > > As I said we need to tidy up some of the naming and definitions, > > > > but > > > technically I don't see any problems - we designed it to work this > way. > > > > > > > > > > For me, this is not at all clear. In fact, the latest info model > > > also has a status channel and a logging channel. It is great if we > > > can ship stuff anywhere - but I think we would be even better off > if > > > we have a clear view who should be the receiver / consumer of > status > > > and logging information. I think this is actually an architectural > > > question to answer. > > > > > > /js > > > > > > -- > > > Juergen Schoenwaelder Jacobs University Bremen gGmbH > > > Phone: +49 421 200 3587 Campus Ring 1, 28759 Bremen, > Germany > > > Fax: +49 421 200 3103 > > > _______________________________________________ > > > lmap mailing list > > > lmap@ietf.org > > > https://www.ietf.org/mailman/listinfo/lmap >=20 > -- > Juergen Schoenwaelder Jacobs University Bremen gGmbH > Phone: +49 421 200 3587 Campus Ring 1, 28759 Bremen, Germany > Fax: +49 421 200 3103 From j.schoenwaelder@jacobs-university.de Fri Nov 29 04:01:31 2013 Return-Path: X-Original-To: lmap@ietfa.amsl.com Delivered-To: lmap@ietfa.amsl.com Received: from localhost (ietfa.amsl.com [127.0.0.1]) by ietfa.amsl.com (Postfix) with ESMTP id EEBF11ADF4E for ; Fri, 29 Nov 2013 04:01:30 -0800 (PST) X-Virus-Scanned: amavisd-new at amsl.com X-Spam-Flag: NO X-Spam-Score: -2.251 X-Spam-Level: X-Spam-Status: No, score=-2.251 tagged_above=-999 required=5 tests=[BAYES_00=-1.9, HELO_EQ_DE=0.35, RCVD_IN_DNSWL_LOW=-0.7, RP_MATCHES_RCVD=-0.001] autolearn=ham Received: from mail.ietf.org ([4.31.198.44]) by localhost (ietfa.amsl.com [127.0.0.1]) (amavisd-new, port 10024) with ESMTP id ruHypw3lXSbQ for ; Fri, 29 Nov 2013 04:01:28 -0800 (PST) Received: from hermes.jacobs-university.de (hermes.jacobs-university.de [212.201.44.23]) by ietfa.amsl.com (Postfix) with ESMTP id 279DF1ADF34 for ; Fri, 29 Nov 2013 04:01:28 -0800 (PST) Received: from localhost (demetrius2.jacobs-university.de [212.201.44.47]) by hermes.jacobs-university.de (Postfix) with ESMTP id 7299E2008A; Fri, 29 Nov 2013 13:01:26 +0100 (CET) X-Virus-Scanned: amavisd-new at jacobs-university.de Received: from hermes.jacobs-university.de ([212.201.44.23]) by localhost (demetrius2.jacobs-university.de [212.201.44.32]) (amavisd-new, port 10024) with ESMTP id jXOmpSXYJOhS; Fri, 29 Nov 2013 13:01:26 +0100 (CET) Received: from elstar.local (elstar.jacobs.jacobs-university.de [10.50.231.133]) by hermes.jacobs-university.de (Postfix) with ESMTP id AD85020084; Fri, 29 Nov 2013 13:01:25 +0100 (CET) Received: by elstar.local (Postfix, from userid 501) id 4F7AF298C6A0; Fri, 29 Nov 2013 13:01:19 +0100 (CET) Date: Fri, 29 Nov 2013 13:01:19 +0100 From: Juergen Schoenwaelder To: philip.eardley@bt.com Message-ID: <20131129120118.GA72776@elstar.local> Mail-Followup-To: philip.eardley@bt.com, trevor.burbridge@bt.com, steve@idrathernotsay.com, lmap@ietf.org References: <20131128164756.GA70319@elstar.local> MIME-Version: 1.0 Content-Type: text/plain; charset=us-ascii Content-Disposition: inline In-Reply-To: User-Agent: Mutt/1.5.21 (2010-09-15) Cc: trevor.burbridge@bt.com, steve@idrathernotsay.com, lmap@ietf.org Subject: Re: [lmap] Defining Status and Logging (Capabilities and Failure) X-BeenThere: lmap@ietf.org X-Mailman-Version: 2.1.15 Precedence: list Reply-To: Juergen Schoenwaelder List-Id: Large Scale Measurement of Access network Performance List-Unsubscribe: , List-Archive: List-Post: List-Help: List-Subscribe: , X-List-Received-Date: Fri, 29 Nov 2013 12:01:31 -0000 On Fri, Nov 29, 2013 at 10:56:05AM +0000, philip.eardley@bt.com wrote: > Yes, it's always possible to report info to the Collector and from there (via an out-of-scope mechanism) to the Controller. The example we gave was the results of one test leading to some follow-up tests. Maybe this type of interaction should be assumed where we have to analyse the results before deciding what Task to do next - if you're trying to diagnose a fault. > Whereas if you're trying to decide how big a file to use for your download speed test, that depends on operational state (the interface speed) You can design this into the test itself by making the test adapt to the interface speed and perhaps also at the statistical properties of the download, e.g. you request a big file but abort the transfer once you have sufficient data received. The more self-contained tests can be, the less the controller needs to understand about every test, which leads to a great simplification overall. > The info model has lots of optional stuff concerning operational state. Who decides what's sent? Does the Controller's Request-for-operational-info include the list of things it wants to learn? Or not - and then does the MA reply with all Operational-info? Everything that's changed since it last sent? Any operational-info at its choice? > Do we allow extensibility? Ie can the operator of the measurement system include its own additional operational-info fields? If yes, how? > > My guesses would be: > Request-for-operational-info(list) or (all) > Can define own operator /vendor specific fields I think it is fair to assume that there is a mechanism that allows a controller to select the operational state it is interested in. Whether there is an "everything since last time" filter, I do not know - this requires some state to be kept on the device. And yes, I believe operational state needs to be extensible and hence I suggested to actually reduce stuff we currently have in the information model so that we do not spend lots of time discussing what operational state might all be useful (since this depends to a large extend on the tests you are running). /js -- Juergen Schoenwaelder Jacobs University Bremen gGmbH Phone: +49 421 200 3587 Campus Ring 1, 28759 Bremen, Germany Fax: +49 421 200 3103 From steve@idrathernotsay.com Sat Nov 30 14:07:14 2013 Return-Path: X-Original-To: lmap@ietfa.amsl.com Delivered-To: lmap@ietfa.amsl.com Received: from localhost (ietfa.amsl.com [127.0.0.1]) by ietfa.amsl.com (Postfix) with ESMTP id 41B6A1AE49A for ; Sat, 30 Nov 2013 14:07:14 -0800 (PST) X-Virus-Scanned: amavisd-new at amsl.com X-Spam-Flag: NO X-Spam-Score: 0.597 X-Spam-Level: X-Spam-Status: No, score=0.597 tagged_above=-999 required=5 tests=[BAYES_40=-0.001, J_CHICKENPOX_46=0.6, RP_MATCHES_RCVD=-0.001, SPF_PASS=-0.001] autolearn=no Received: from mail.ietf.org ([4.31.198.44]) by localhost (ietfa.amsl.com [127.0.0.1]) (amavisd-new, port 10024) with ESMTP id tMazj5KfJzWY for ; Sat, 30 Nov 2013 14:07:12 -0800 (PST) Received: from colo3.idrathernotsay.com (colo3.idrathernotsay.com [74.207.251.199]) by ietfa.amsl.com (Postfix) with ESMTP id CB8A71AE499 for ; Sat, 30 Nov 2013 14:07:12 -0800 (PST) Received: from localhost (localhost [127.0.0.1]) by colo3.idrathernotsay.com (Postfix) with ESMTP id C0800FC36; Sat, 30 Nov 2013 22:07:10 +0000 (UTC) X-Virus-Scanned: amavisd-new at idrathernotsay.com Received: from colo3.idrathernotsay.com ([127.0.0.1]) by localhost (colo3.idrathernotsay.com [127.0.0.1]) (amavisd-new, port 10024) with ESMTP id 5bBjtw1YOxKv; Sat, 30 Nov 2013 22:07:09 +0000 (UTC) Received: from misc-185.idrathernotsay.com (pool-173-79-225-215.washdc.fios.verizon.net [173.79.225.215]) by colo3.idrathernotsay.com (Postfix) with ESMTPSA id 2355EFBFA; Sat, 30 Nov 2013 22:07:09 +0000 (UTC) Message-ID: <529A618C.4010509@idrathernotsay.com> Date: Sat, 30 Nov 2013 17:07:08 -0500 From: Steven Miller User-Agent: Mozilla/5.0 (Macintosh; Intel Mac OS X 10.7; rv:24.0) Gecko/20100101 Thunderbird/24.0.1 MIME-Version: 1.0 To: philip.eardley@bt.com, lmap@ietf.org References: <20131111182839.GA21454@idrathernotsay.com> <20131111204218.GB770@elstar.local> <20131112185611.GA33370@idrathernotsay.com> <20131128070910.GB68751@elstar.local> In-Reply-To: <20131128070910.GB68751@elstar.local> Content-Type: text/plain; charset=ISO-8859-1; format=flowed Content-Transfer-Encoding: 7bit Subject: Re: [lmap] Splitting measurement tasks into measurement tasks plus reporting tasks? X-BeenThere: lmap@ietf.org X-Mailman-Version: 2.1.15 Precedence: list List-Id: Large Scale Measurement of Access network Performance List-Unsubscribe: , List-Archive: List-Post: List-Help: List-Subscribe: , X-List-Received-Date: Sat, 30 Nov 2013 22:07:14 -0000 Funny that I just sent Phil private mail where I described the heartbeat system (and related auto-shutdown logic) we'd put in place and how it proved largely to be much more trouble than it was worth... Our stuff shut down after 10 minutes or so of not hearing from our equivalent of a controller, on the theory that maybe we'd botched a measurement definition and eaten all the bandwidth. Until we cranked that particular timer it was pretty easy for us to get false positives. I wouldn't be averse to something taking itself offline if it's gone for an extended period without reaching its controller, or at least maybe after a few days -- longer than a typical service outage? -- it stops doing tests until it succeeds in reaching the controller. -Steve On 11/28/2013 2:09 AM, Juergen Schoenwaelder wrote: > Hi, > > I think instructions must be considered some sort of soft state in > general. An MA that for whatever reason looses connectivity to the > controller for a very long time or permanently should stop doing > measurements at some point in time and not continue indefinitely. > > Concerning the second point, I do not really see a problem with > allowing a measurement task to create multiple result records. I can > see that this is useful and easy to implement. I do not see how not > allowing this would simplify anything. > > /js > > On Wed, Nov 27, 2013 at 05:00:39PM +0000, philip.eardley@bt.com wrote: >> I take two questions from this discussion: >> >> 1. can a Measurement Task run indefinitely? >> 2. can a single Measurement Task create more than one Measurement Report? (for instance, one the long term average, the other an immediate alarm that a parameter has gone below a threshold, say) >> >> 1. I think this just needs some explanatory text on these lines: >> The Controller may want a MA to run the same Measurement Task indefinitely (for example, "run the 'upload speed' Measurement Task once an hour until further notice"). To avoid the MA generating traffic forever after a Controller has permanently failed, it is suggested that the Measurement Schedule includes a time limit ("run the 'upload speed' Measurement Task once an hour for the next 30 days") and that the Measurement Schedule is updated regularly (say, every 10 days). >> >> Does this work? >> >> 2. think this needs a bit more discussion. I can see it might be useful (if we have a strict 1:1 Task:Result, then the example above would need 2 Tasks and hence 2 lots of data to be sent. For some Tasks this would be ok, for others not). So I'm OK if people think it won't add too much complexity to the Information Model and Protocol. >> >> (Incidentally this is in tension with the email "Only Report raw measurement results". Perhaps we could instead describe this as two linked Tasks (each with one Result) which in fact create only one pkt stream; perhaps this is just confusing.) >> >> Thoughts? >> >> From steve@idrathernotsay.com Sat Nov 30 14:22:08 2013 Return-Path: X-Original-To: lmap@ietfa.amsl.com Delivered-To: lmap@ietfa.amsl.com Received: from localhost (ietfa.amsl.com [127.0.0.1]) by ietfa.amsl.com (Postfix) with ESMTP id 26AE21AE49C for ; Sat, 30 Nov 2013 14:22:08 -0800 (PST) X-Virus-Scanned: amavisd-new at amsl.com X-Spam-Flag: NO X-Spam-Score: -1.902 X-Spam-Level: X-Spam-Status: No, score=-1.902 tagged_above=-999 required=5 tests=[BAYES_00=-1.9, RP_MATCHES_RCVD=-0.001, SPF_PASS=-0.001] autolearn=ham Received: from mail.ietf.org ([4.31.198.44]) by localhost (ietfa.amsl.com [127.0.0.1]) (amavisd-new, port 10024) with ESMTP id XuWekt4xu7pY for ; Sat, 30 Nov 2013 14:22:06 -0800 (PST) Received: from colo3.idrathernotsay.com (colo3.idrathernotsay.com [74.207.251.199]) by ietfa.amsl.com (Postfix) with ESMTP id 1DBA41AE48B for ; Sat, 30 Nov 2013 14:22:06 -0800 (PST) Received: from localhost (localhost [127.0.0.1]) by colo3.idrathernotsay.com (Postfix) with ESMTP id B876DFC36; Sat, 30 Nov 2013 22:22:04 +0000 (UTC) X-Virus-Scanned: amavisd-new at idrathernotsay.com Received: from colo3.idrathernotsay.com ([127.0.0.1]) by localhost (colo3.idrathernotsay.com [127.0.0.1]) (amavisd-new, port 10024) with ESMTP id W1TYigvkGoY1; Sat, 30 Nov 2013 22:22:02 +0000 (UTC) Received: from misc-185.idrathernotsay.com (pool-173-79-225-215.washdc.fios.verizon.net [173.79.225.215]) by colo3.idrathernotsay.com (Postfix) with ESMTPSA id 02DEDFBFA; Sat, 30 Nov 2013 22:22:01 +0000 (UTC) Message-ID: <529A6508.2020706@idrathernotsay.com> Date: Sat, 30 Nov 2013 17:22:00 -0500 From: Steven Miller User-Agent: Mozilla/5.0 (Macintosh; Intel Mac OS X 10.7; rv:24.0) Gecko/20100101 Thunderbird/24.0.1 MIME-Version: 1.0 To: philip.eardley@bt.com, trevor.burbridge@bt.com, lmap@ietf.org, j.schoenwaelder@jacobs-university.de References: <20131128164756.GA70319@elstar.local> <20131129120118.GA72776@elstar.local> In-Reply-To: <20131129120118.GA72776@elstar.local> Content-Type: text/plain; charset=ISO-8859-1; format=flowed Content-Transfer-Encoding: 7bit Subject: Re: [lmap] Defining Status and Logging (Capabilities and Failure) X-BeenThere: lmap@ietf.org X-Mailman-Version: 2.1.15 Precedence: list List-Id: Large Scale Measurement of Access network Performance List-Unsubscribe: , List-Archive: List-Post: List-Help: List-Subscribe: , X-List-Received-Date: Sat, 30 Nov 2013 22:22:08 -0000 FWIW I like the idea that feeding operational state back is just another measurement task. The stuff in the current I-D does seem like a long list, though as I read it with an eye to things I thought could easily be thrown out, I found myself thinking that most or all of that is stuff I'd like to have available if I was trying to figure out after the fact why something had started performing poorly. For example, knowing the gateway that was in use at a time more-or-less around when a given measurement started reporting poor results might let me cross-correlate that back to a device for which I have some other failure information, a la "oh, that was this router, yeah, there was a device-down alert for that at the same time, that explains it." -Steve On 11/29/2013 7:01 AM, Juergen Schoenwaelder wrote: > On Fri, Nov 29, 2013 at 10:56:05AM +0000, philip.eardley@bt.com wrote: > >> Yes, it's always possible to report info to the Collector and from there (via an out-of-scope mechanism) to the Controller. The example we gave was the results of one test leading to some follow-up tests. Maybe this type of interaction should be assumed where we have to analyse the results before deciding what Task to do next - if you're trying to diagnose a fault. >> Whereas if you're trying to decide how big a file to use for your download speed test, that depends on operational state (the interface speed) > You can design this into the test itself by making the test adapt to > the interface speed and perhaps also at the statistical properties of > the download, e.g. you request a big file but abort the transfer once > you have sufficient data received. The more self-contained tests can > be, the less the controller needs to understand about every test, > which leads to a great simplification overall. > >> The info model has lots of optional stuff concerning operational state. Who decides what's sent? Does the Controller's Request-for-operational-info include the list of things it wants to learn? Or not - and then does the MA reply with all Operational-info? Everything that's changed since it last sent? Any operational-info at its choice? >> Do we allow extensibility? Ie can the operator of the measurement system include its own additional operational-info fields? If yes, how? >> >> My guesses would be: >> Request-for-operational-info(list) or (all) >> Can define own operator /vendor specific fields > I think it is fair to assume that there is a mechanism that allows a > controller to select the operational state it is interested > in. Whether there is an "everything since last time" filter, I do not > know - this requires some state to be kept on the device. And yes, I > believe operational state needs to be extensible and hence I suggested > to actually reduce stuff we currently have in the information model so > that we do not spend lots of time discussing what operational state > might all be useful (since this depends to a large extend on the tests > you are running). > > /js >