Subject: RE: [ippm] Application loss metrics

From: "Hedayat, Kaynam" <[email-masked]>
Date: Wed, 28 Feb 2007 16:57:36 -0500

The purpose of this draft is not to introduce an objective video quality algorithm. Rather it is to define "yardstick" metrics that are well-defined and are not tied to subjective data. I agree with Alan that there is a lot of work in ITU-T and VQEG to define objective metrics in this area. However, I caution the industry both in terms of the timeline for availability of such metrics and general applicability of such metrics to all networks and for all types of monitoring applications. In the meantime the industry needs "yardstick" metrics that can be used to effectively monitor the performance of IP networks.

I encourage everyone to look for guidance from ITU and VQEG for video quality algorithms and not industry forums such as ATIS and DSL Forum. It is the job of ITU and VQEG to standardize the algorithm and job of ATIS, DSL Forum, etc. to define the applicability of such algorithms to specific networks and applications. The expertise within the standard bodies and industry forums is based on this philosophy.

As for the applicability of proposed metrics to video and voice, they have direct correlation with voice and video impairments.

Regards,
Kaynam

> -----Original Message-----
> From: Alan Clark [mailto:[email-masked]] 
> Sent: Tuesday, February 27, 2007 8:20 AM
> To: 'Geib, Ruediger'; Venna, Nagarjuna
> Cc: [email-masked]
> Subject: RE: [ippm] Application loss metrics
> 
> Ruediger
> 
> There is considerable work within the standards community in 
> the development of objective video measurement techniques, 
> much of which is fairly advanced.
> Some of the lead groups include ITU-T Study Group 9 and 12, 
> the ITU-T IPTV Focus Group, the Video Quality Experts Group 
> (VQEG), ATIS, DSL Forum, TM Forum.........  
> Within IETF, AVT have a working group draft - 
> draft-ietf-avt-rtcpxr-video-00.txt - which has been under 
> development for the last year, this focusses more on the 
> transport of metrics rather than defining new video objective 
> techniques.
> 
> Objective video quality estimation algorithms are not east to test
> (objectively) and hence the development of test programs and 
> the work of comparison with subjective test data often 
> becomes complex.  This is one reason that VQEG exists - it 
> provides a testing function that works with
> many other standards organizations.   
> 
> With regard to your specific observations on I/B/P frames - 
> this is absolutely considered within the work done by the 
> various ITU groups and ATIS.  One technique that has been 
> defined, discussed across multiple standards organizations 
> and is currently being assessed is based on PSNR estimation - 
> which is taking into account loss propagation effects through 
> the GoP as well as other coding parameters.  
> 
> My recommendation would be that IPPM does not start work in 
> developing objective video quality estimation algorithms, as 
> this duplicates work already underway in a number of other 
> groups and IPPM may not have the expertize in this area.
> 
> IPPM may note however that there are some "holes" in the IPPM 
> metrics as they apply to voice and video over IP.  Packet 
> loss often occurs in sparse bursts, for example periods of 
> 5-30% loss that can last for periods of up to several 
> seconds, and IPPM don't have a metric to describe these.
> 
> 
> Regards
> 
> Alan Clark
> Telchemy
> 
>  
> 
> -----Original Message-----
> From: Geib, Ruediger [mailto:[email-masked]]
> Sent: Tuesday, February 27, 2007 3:17 AM
> To: [email-masked]
> Cc: [email-masked]
> Subject: RE: [ippm] Application loss metrics
> 
> Dear Nagarjuna
> 
> your proposal to work on application level quality metrics is 
> allright with me. To the best of my knowledge, up to now 
> there's no objective IP TV/video conferencing quality 
> measurement based on subjective judgements which would 
> correspond e.g. to those models available for telephony services.
> 
> Considering bare "frame loss" however may not result in a 
> satisfactory judgement of the influence of loss on a 
> streaming application. It does not
> characterise:
> 
> - the importance of the frame which was lost within a
>   Group of Pictures. I frames carry full pictures and
>   serve as a reference, P frames serve as a reference,
>   whereas B frames don't serve as a reference to
>   other frames.
> 
> - the impact of the frame drop within a Group of
>   Pictures on the error propagation within a group of
>   pictures. If a part of an I frame is lost, all other
>   frames referrring to the lost section will display
>   errouneous information no matter whether they are
>   received correctly or not. On the other side, if
>   just a B-frame is lost, only a single picture
>   is influenced (and not a complete Group of
>   Pictures).
> 
> - the different size of frames in terms of IP packets
>   per frame. An I frame may be bigger than a P frame. 
>   Both definitely are significantly bigger than
>   B frames.
> 
> For those who aren't familiar with streaming: A Group of 
> Pictures is the basic way of tranmistting a compressed 
> picture by one complete picture and a set of differential 
> changes to it. Each frame results in an own picture and the 
> Group of Picture is:
> 
> I - BB - P - BB - P.... until next I. Not all frame types 
> must be present (except I frames).
> 
> Regards,
> 
> Ruediger
> 
> -----Original Message-----
> From: Venna, Nagarjuna [mailto:[email-masked]]
> Sent: Tuesday, February 27, 2007 4:43 AM
> To: [email-masked]
> Subject: [ippm] Application loss metrics
> 
> 
> 
> 
> I have posted an individual draft
> (http://www.ietf.org/internet-drafts/draft-venna-ippm-app-loss
> -metrics-00.tx
> t) that describes the application level metrics. These are 
> the same metrics I have talked about in the San Diego meeting.
> 
> Comments and feedback are appreciated.
> 
> Thanks,
> Nagarjuna 
> 
> 
> 
> --------------------------------------------------------------
> --------------
> --------------------------------------------------
> Service Quality Matters. Test the performance and quality of 
> your VoIP or IP video service at:
> http://www.TestYourVoIP.com
> http://www.TestYourIPVideo.com
> 
> ---------------------------------------------------
> 
> Rüdiger Geib
> 
> T-Systems Enterprise Services GmbH
> Systems Integration
> Rüdiger Geib
> Line of Business Networks & Processes
> Senior Consultant
> Deutsche-Telekom-Allee 7, 64295 Darmstadt
> +49 6151 937-2138 (Tel.)
> http://www.t-systems.com
> 
> T-Systems Enterprise Services GmbH
> Aufsichtsrat: René Obermann (Vorsitzender) Executive 
> Committee: Lothar Pauly (Vorsitzender)*, Helmut Binder, 
> Albert Henn, Olaf Heyden*, Katrin Horstmann, Ulrich Kemp, 
> Axel Knobe*, Wilfried Peters*, Dr. Herbert Schaaff*, Zvezdana
> Seeger*
> Handelsregister: Amtsgericht Frankfurt am Main HRB 55933 Sitz der
> Gesellschaft: Frankfurt am Main WEEE-Reg.-Nr. DE87523644 
> *Geschäftsführer gem. § 35 GmbHG
> 
> Notice: This transmittal and/or attachments may be privileged 
> or confidential. If you are not the intended recipient, you 
> are hereby notified that you have received this transmittal 
> in error; any review, dissemination, or copying is strictly 
> prohibited. If you received this transmittal in error, please 
> notify us immediately by reply and immediately delete this 
> message and all its attachments. Thank you.
> 
>  
> 
> _______________________________________________
> ippm mailing list
> [email-masked]
> https://www1.ietf.org/mailman/listinfo/ippm
> 
> 
> _______________________________________________
> ippm mailing list
> [email-masked]
> https://www1.ietf.org/mailman/listinfo/ippm
> 

_______________________________________________
ippm mailing list
[email-masked] 
https://www1.ietf.org/mailman/listinfo/ippm

---

From: "Alan Clark" <[email-masked]>
Date: Wed, 28 Feb 2007 17:51:49 -0500

IMHO, the industry has more IP metrics than network managers can typically
understand, or at least many IP metrics that network managers cannot
understand :-)

It seems somewhat counterintuitive to address the issue of the time taken to
develop objective quality measures by adding new work items to the industry
"desk".  There are also a number of IP metrics that already characterize IP
impairments that affect voice and video performance, for example the
burst/gap metrics in RFC3611 (also in RTCP HR, RTCP XR Video Metrics and
soon to be added to Y.1540).
I also note that you use the expression "yardstick metrics that are
well-defined and are not tied to subjective data" - presumably a yardstick
metric would be most useful if it had some bearing on quality as perceived
by the user?   Having a well defined metric that had no relationship to
subjective quality would appear to be of limited application?

I would also suggest that IPPM consider cooperation with AVT in this area,
as within the IETF the AVT group has established expertize in the voice/
video area and has both completed and current work related to reporting the
impact of IP impairments on voice/video performance.

ATIS committees by the way are not industry fora but regional accredited
standards committees, similar to TIA, ETSI etc. and have a long history of
developing standards related to network performance, quality and a wide
range of telecom service related issues. There are many other standards
organizations that actively work in the area of voice and video performance,
including ITU-R, ETSI STQ....  Whilst your "encouragement" may be well
intentioned and your aim to direct others to the appropriate standards
committees that have established expertize - this may be a disservice as you
may not be familiar with the range of activities that are current within the
industry and the degree of cooperation that exists between the various
groups that are active in the multimedia performance arena.

Regards

Alan

-----Original Message-----
From: Hedayat, Kaynam [mailto:[email-masked]] 
Sent: Wednesday, February 28, 2007 4:58 PM
To: Alan Clark; Geib, Ruediger; Venna, Nagarjuna
Cc: [email-masked]
Subject: RE: [ippm] Application loss metrics

The purpose of this draft is not to introduce an objective video quality
algorithm. Rather it is to define "yardstick" metrics that are well-defined
and are not tied to subjective data. I agree with Alan that there is a lot
of work in ITU-T and VQEG to define objective metrics in this area. However,
I caution the industry both in terms of the timeline for availability of
such metrics and general applicability of such metrics to all networks and
for all types of monitoring applications. In the meantime the industry needs
"yardstick" metrics that can be used to effectively monitor the performance
of IP networks.

I encourage everyone to look for guidance from ITU and VQEG for video
quality algorithms and not industry forums such as ATIS and DSL Forum. It is
the job of ITU and VQEG to standardize the algorithm and job of ATIS, DSL
Forum, etc. to define the applicability of such algorithms to specific
networks and applications. The expertise within the standard bodies and
industry forums is based on this philosophy.

As for the applicability of proposed metrics to video and voice, they have
direct correlation with voice and video impairments.

Regards,
Kaynam

> There is considerable work within the standards community in the 
> development of objective video measurement techniques, much of which 
> is fairly advanced.
> Some of the lead groups include ITU-T Study Group 9 and 12, the ITU-T 
> IPTV Focus Group, the Video Quality Experts Group (VQEG), ATIS, DSL 
> Forum, TM Forum.........
> draft-ietf-avt-rtcpxr-video-00.txt - which has been under development 
> for the last year, this focusses more on the transport of metrics 
> rather than defining new video objective techniques.
> 
> (objectively) and hence the development of test programs and the work 
> of comparison with subjective test data often becomes complex.  This 
> is one reason that VQEG exists - it provides a testing function that 
> works with
> With regard to your specific observations on I/B/P frames - this is 
> absolutely considered within the work done by the various ITU groups 
> and ATIS.  One technique that has been defined, discussed across 
> multiple standards organizations and is currently being assessed is 
> based on PSNR estimation - which is taking into account loss 
> propagation effects through the GoP as well as other coding 
> parameters.
> 
> My recommendation would be that IPPM does not start work in developing 
> objective video quality estimation algorithms, as this duplicates work 
> already underway in a number of other groups and IPPM may not have the 
> expertize in this area.
> 
> IPPM may note however that there are some "holes" in the IPPM metrics 
> as they apply to voice and video over IP.  Packet loss often occurs in 
> sparse bursts, for example periods of 5-30% loss that can last for 
> periods of up to several seconds, and IPPM don't have a metric to 
> describe these.
> 
> 
> Regards
> 
> Alan Clark
> Telchemy
> 
>  
> 
> your proposal to work on application level quality metrics is allright 
> with me. To the best of my knowledge, up to now there's no objective 
> IP TV/video conferencing quality measurement based on subjective 
> judgements which would correspond e.g. to those models available for 
> telephony services.
> 
> Considering bare "frame loss" however may not result in a satisfactory 
> judgement of the influence of loss on a streaming application. It does 
> not
> characterise:
> 
> For those who aren't familiar with streaming: A Group of Pictures is 
> the basic way of tranmistting a compressed picture by one complete 
> picture and a set of differential changes to it. Each frame results in 
> an own picture and the Group of Picture is:
> 
> I - BB - P - BB - P.... until next I. Not all frame types must be 
> present (except I frames).
> 
> Regards,
> 
> Ruediger
> 
> t) that describes the application level metrics. These are the same 
> metrics I have talked about in the San Diego meeting.
> 
> Service Quality Matters. Test the performance and quality of your VoIP 
> or IP video service at:
> Committee: Lothar Pauly (Vorsitzender)*, Helmut Binder, Albert Henn, 
> Olaf Heyden*, Katrin Horstmann, Ulrich Kemp, Axel Knobe*, Wilfried 
> Peters*, Dr. Herbert Schaaff*, Zvezdana
> Seeger*
> Notice: This transmittal and/or attachments may be privileged or 
> confidential. If you are not the intended recipient, you are hereby 
> notified that you have received this transmittal in error; any review, 
> dissemination, or copying is strictly prohibited. If you received this 
> transmittal in error, please notify us immediately by reply and 
> immediately delete this message and all its attachments. Thank you.
> 
>  
>