Subject: RE: [ippm] Application loss metrics

From: "Geib, Ruediger" <[email-masked]>
Date: Thu, 1 Mar 2007 08:36:12 +0100

Kaynam,

|As for the applicability of proposed metrics to video and 
|voice, they have direct correlation with voice and video 
|impairments.

Your current draft doesn't contain information on measurements 
or simulations proving this claim, nor does it refer to any 
publication serving this purpose. Please provide some
references supporting your statement.

Regards,

Ruediger

|> -----Original Message-----
|> From: Alan Clark [mailto:[email-masked]] 
|> Sent: Tuesday, February 27, 2007 8:20 AM
|> To: 'Geib, Ruediger'; Venna, Nagarjuna
|> Cc: [email-masked]
|> Subject: RE: [ippm] Application loss metrics
|> 
|> Ruediger
|> 
|> There is considerable work within the standards community in 
|> the development of objective video measurement techniques, 
|> much of which is fairly advanced.
|> Some of the lead groups include ITU-T Study Group 9 and 12, 
|> the ITU-T IPTV Focus Group, the Video Quality Experts Group 
|> (VQEG), ATIS, DSL Forum, TM Forum.........  
|> Within IETF, AVT have a working group draft - 
|> draft-ietf-avt-rtcpxr-video-00.txt - which has been under 
|> development for the last year, this focusses more on the 
|> transport of metrics rather than defining new video objective 
|> techniques.
|> 
|> Objective video quality estimation algorithms are not east to test
|> (objectively) and hence the development of test programs and 
|> the work of comparison with subjective test data often 
|> becomes complex.  This is one reason that VQEG exists - it 
|> provides a testing function that works with
|> many other standards organizations.   
|> 
|> With regard to your specific observations on I/B/P frames - 
|> this is absolutely considered within the work done by the 
|> various ITU groups and ATIS.  One technique that has been 
|> defined, discussed across multiple standards organizations 
|> and is currently being assessed is based on PSNR estimation - 
|> which is taking into account loss propagation effects through 
|> the GoP as well as other coding parameters.  
|> 
|> My recommendation would be that IPPM does not start work in 
|> developing objective video quality estimation algorithms, as 
|> this duplicates work already underway in a number of other 
|> groups and IPPM may not have the expertize in this area.
|> 
|> IPPM may note however that there are some "holes" in the IPPM 
|> metrics as they apply to voice and video over IP.  Packet 
|> loss often occurs in sparse bursts, for example periods of 
|> 5-30% loss that can last for periods of up to several 
|> seconds, and IPPM don't have a metric to describe these.
|> 
|> 
|> Regards
|> 
|> Alan Clark
|> Telchemy
|> 
|>  
|> 
|> -----Original Message-----
|> From: Geib, Ruediger [mailto:[email-masked]]
|> Sent: Tuesday, February 27, 2007 3:17 AM
|> To: [email-masked]
|> Cc: [email-masked]
|> Subject: RE: [ippm] Application loss metrics
|> 
|> Dear Nagarjuna
|> 
|> your proposal to work on application level quality metrics is 
|> allright with me. To the best of my knowledge, up to now 
|> there's no objective IP TV/video conferencing quality 
|> measurement based on subjective judgements which would 
|> correspond e.g. to those models available for telephony services.
|> 
|> Considering bare "frame loss" however may not result in a 
|> satisfactory judgement of the influence of loss on a 
|> streaming application. It does not
|> characterise:
|> 
|> - the importance of the frame which was lost within a
|>   Group of Pictures. I frames carry full pictures and
|>   serve as a reference, P frames serve as a reference,
|>   whereas B frames don't serve as a reference to
|>   other frames.
|> 
|> - the impact of the frame drop within a Group of
|>   Pictures on the error propagation within a group of
|>   pictures. If a part of an I frame is lost, all other
|>   frames referrring to the lost section will display
|>   errouneous information no matter whether they are
|>   received correctly or not. On the other side, if
|>   just a B-frame is lost, only a single picture
|>   is influenced (and not a complete Group of
|>   Pictures).
|> 
|> - the different size of frames in terms of IP packets
|>   per frame. An I frame may be bigger than a P frame. 
|>   Both definitely are significantly bigger than
|>   B frames.
|> 
|> For those who aren't familiar with streaming: A Group of 
|> Pictures is the basic way of tranmistting a compressed 
|> picture by one complete picture and a set of differential 
|> changes to it. Each frame results in an own picture and the 
|> Group of Picture is:
|> 
|> I - BB - P - BB - P.... until next I. Not all frame types 
|> must be present (except I frames).
|> 
|> Regards,
|> 
|> Ruediger
|> 
|> -----Original Message-----
|> From: Venna, Nagarjuna [mailto:[email-masked]]
|> Sent: Tuesday, February 27, 2007 4:43 AM
|> To: [email-masked]
|> Subject: [ippm] Application loss metrics
|> 
|> 
|> 
|> 
|> I have posted an individual draft
|> (http://www.ietf.org/internet-drafts/draft-venna-ippm-app-loss
|> -metrics-00.tx
|> t) that describes the application level metrics. These are 
|> the same metrics I have talked about in the San Diego meeting.
|> 
|> Comments and feedback are appreciated.
|> 
|> Thanks,
|> Nagarjuna 
|> 
|> 
|> 
|> --------------------------------------------------------------
|> --------------
|> --------------------------------------------------
|> Service Quality Matters. Test the performance and quality of 
|> your VoIP or IP video service at:
|> http://www.TestYourVoIP.com
|> http://www.TestYourIPVideo.com
|> 
|> ---------------------------------------------------
|> 
|> Rüdiger Geib
|> 
|> T-Systems Enterprise Services GmbH
|> Systems Integration
|> Rüdiger Geib
|> Line of Business Networks & Processes
|> Senior Consultant
|> Deutsche-Telekom-Allee 7, 64295 Darmstadt
|> +49 6151 937-2138 (Tel.)
|> http://www.t-systems.com
|> 
|> T-Systems Enterprise Services GmbH
|> Aufsichtsrat: René Obermann (Vorsitzender) Executive 
|> Committee: Lothar Pauly (Vorsitzender)*, Helmut Binder, 
|> Albert Henn, Olaf Heyden*, Katrin Horstmann, Ulrich Kemp, 
|> Axel Knobe*, Wilfried Peters*, Dr. Herbert Schaaff*, Zvezdana
|> Seeger*
|> Handelsregister: Amtsgericht Frankfurt am Main HRB 55933 Sitz der
|> Gesellschaft: Frankfurt am Main WEEE-Reg.-Nr. DE87523644 
|> *Geschäftsführer gem. § 35 GmbHG
|> 
|> Notice: This transmittal and/or attachments may be privileged 
|> or confidential. If you are not the intended recipient, you 
|> are hereby notified that you have received this transmittal 
|> in error; any review, dissemination, or copying is strictly 
|> prohibited. If you received this transmittal in error, please 
|> notify us immediately by reply and immediately delete this 
|> message and all its attachments. Thank you.
|> 
|>  
|> 
|> _______________________________________________
|> ippm mailing list
|> [email-masked]
|> https://www1.ietf.org/mailman/listinfo/ippm
|> 
|> 
|> _______________________________________________
|> ippm mailing list
|> [email-masked]
|> https://www1.ietf.org/mailman/listinfo/ippm
|> 
|
|_______________________________________________
|ippm mailing list
[email-masked] 
|https://www1.ietf.org/mailman/listinfo/ippm
|

_______________________________________________
ippm mailing list
[email-masked] 
https://www1.ietf.org/mailman/listinfo/ippm

---

From: "Hedayat, Kaynam" <[email-masked]>
Date: Thu, 1 Mar 2007 10:44:35 -0500

Ruediger,

I did not mean to state that an exact correlation or formula exists between these metrics and video impairments. The intend of the draft is simply to provide metrics that indicate problems in the network that may cause video impairments. We have found, based on our experience, that metrics defined in rfc3357 apply to the application layer.

Regards,
Kaynam

> -----Original Message-----
> From: Geib, Ruediger [mailto:[email-masked]] 
> Sent: Thursday, March 01, 2007 2:36 AM
> To: Hedayat, Kaynam
> Cc: [email-masked]
> Subject: RE: [ippm] Application loss metrics
> 
> Kaynam,
> 
> voice, they 
> |have direct correlation with voice and video impairments.
> 
> Your current draft doesn't contain information on 
> measurements or simulations proving this claim, nor does it 
> refer to any publication serving this purpose. Please provide 
> some references supporting your statement.
> 
> Regards,
> 
> Ruediger
> 
> |> There is considerable work within the standards community in the 
> |> development of objective video measurement techniques, 
> much of which 
> |> is fairly advanced.
> |> Some of the lead groups include ITU-T Study Group 9 and 
> 12, the ITU-T 
> |> IPTV Focus Group, the Video Quality Experts Group (VQEG), 
> ATIS, DSL 
> |> Forum, TM Forum.........
> development 
> |> for the last year, this focusses more on the transport of metrics 
> |> rather than defining new video objective techniques.
> |> 
> |> (objectively) and hence the development of test programs 
> and the work 
> |> of comparison with subjective test data often becomes 
> complex.  This 
> |> is one reason that VQEG exists - it provides a testing 
> function that 
> |> works with
> |> With regard to your specific observations on I/B/P frames 
> - this is 
> |> absolutely considered within the work done by the various 
> ITU groups 
> |> and ATIS.  One technique that has been defined, discussed across 
> |> multiple standards organizations and is currently being 
> assessed is 
> |> based on PSNR estimation - which is taking into account loss 
> |> propagation effects through the GoP as well as other coding 
> |> parameters.
> |> 
> |> developing objective video quality estimation algorithms, as this 
> |> duplicates work already underway in a number of other 
> groups and IPPM 
> |> may not have the expertize in this area.
> |> 
> |> IPPM may note however that there are some "holes" in the 
> IPPM metrics 
> |> as they apply to voice and video over IP.  Packet loss 
> often occurs 
> |> in sparse bursts, for example periods of 5-30% loss that 
> can last for 
> |> periods of up to several seconds, and IPPM don't have a metric to 
> |> describe these.
> |> 
> |> 
> |> Regards
> |> 
> |> Alan Clark
> |> Telchemy
> |> 
> |>  
> |> 
> there's no 
> |> objective IP TV/video conferencing quality measurement based on 
> |> subjective judgements which would correspond e.g. to those models 
> |> available for telephony services.
> |> 
> |> satisfactory judgement of the influence of loss on a streaming 
> |> application. It does not
> |> characterise:
> |> 
> Pictures is 
> |> the basic way of tranmistting a compressed picture by one complete 
> |> picture and a set of differential changes to it. Each 
> frame results 
> |> in an own picture and the Group of Picture is:
> |> 
> |> I - BB - P - BB - P.... until next I. Not all frame types must be 
> |> present (except I frames).
> |> 
> |> Regards,
> |> 
> |> Ruediger
> |> 
> the same 
> |> metrics I have talked about in the San Diego meeting.
> |> 
> |> Service Quality Matters. Test the performance and quality of your 
> |> VoIP or IP video service at:
> |> Line of Business Networks & Processes Senior Consultant 
> Albert Henn, 
> |> Olaf Heyden*, Katrin Horstmann, Ulrich Kemp, Axel Knobe*, Wilfried 
> |> Peters*, Dr. Herbert Schaaff*, Zvezdana
> |> Seeger*
> |> Notice: This transmittal and/or attachments may be privileged or 
> |> confidential. If you are not the intended recipient, you 
> are hereby 
> |> notified that you have received this transmittal in error; any 
> |> review, dissemination, or copying is strictly prohibited. If you 
> |> received this transmittal in error, please notify us 
> immediately by 
> |> reply and immediately delete this message and all its attachments. 
> |> Thank you.
> |> 
> |>  
> |>

---

From: "Geib, Ruediger" <[email-masked]>
Date: Fri, 2 Mar 2007 11:51:18 +0100

Kaynam,

well, my personal view would be that if rfc3557 is applicable 
to indicate problems in the network that may cause application 
impairments, then rfc3557 is sufficient to provide a coarse 
measure. Taking an operative or engineering point of view,
I personally don't see a point in work on more coarse 
measures on possible application layer impairments. 

Regards,

Ruediger

|I did not mean to state that an exact correlation or formula 
|exists between these metrics and video impairments. The intend 
|of the draft is simply to provide metrics that indicate 
|problems in the network that may cause video impairments. We 
|have found, based on our experience, that metrics defined in 
|rfc3357 apply to the application layer.
|
|Regards,
|Kaynam
|
|> -----Original Message-----
|> From: Geib, Ruediger [mailto:[email-masked]] 
|> Sent: Thursday, March 01, 2007 2:36 AM
|> To: Hedayat, Kaynam
|> Cc: [email-masked]
|> Subject: RE: [ippm] Application loss metrics
|> 
|> Kaynam,
|> 
|> |As for the applicability of proposed metrics to video and 
|> voice, they 
|> |have direct correlation with voice and video impairments.
|> 
|> Your current draft doesn't contain information on 
|> measurements or simulations proving this claim, nor does it 
|> refer to any publication serving this purpose. Please provide 
|> some references supporting your statement.
|> 
|> Regards,
|> 
|> Ruediger
|> 
|> |> -----Original Message-----
|> |> From: Alan Clark [mailto:[email-masked]]
|> |> Sent: Tuesday, February 27, 2007 8:20 AM
|> |> To: 'Geib, Ruediger'; Venna, Nagarjuna
|> |> Cc: [email-masked]
|> |> Subject: RE: [ippm] Application loss metrics
|> |> 
|> |> Ruediger
|> |> 
|> |> There is considerable work within the standards community in the 
|> |> development of objective video measurement techniques, 
|> much of which 
|> |> is fairly advanced.
|> |> Some of the lead groups include ITU-T Study Group 9 and 
|> 12, the ITU-T 
|> |> IPTV Focus Group, the Video Quality Experts Group (VQEG), 
|> ATIS, DSL 
|> |> Forum, TM Forum.........
|> |> Within IETF, AVT have a working group draft - 
|> |> draft-ietf-avt-rtcpxr-video-00.txt - which has been under 
|> development 
|> |> for the last year, this focusses more on the transport of metrics 
|> |> rather than defining new video objective techniques.
|> |> 
|> |> Objective video quality estimation algorithms are not east to test
|> |> (objectively) and hence the development of test programs 
|> and the work 
|> |> of comparison with subjective test data often becomes 
|> complex.  This 
|> |> is one reason that VQEG exists - it provides a testing 
|> function that 
|> |> works with
|> |> many other standards organizations.   
|> |> 
|> |> With regard to your specific observations on I/B/P frames 
|> - this is 
|> |> absolutely considered within the work done by the various 
|> ITU groups 
|> |> and ATIS.  One technique that has been defined, discussed across 
|> |> multiple standards organizations and is currently being 
|> assessed is 
|> |> based on PSNR estimation - which is taking into account loss 
|> |> propagation effects through the GoP as well as other coding 
|> |> parameters.
|> |> 
|> |> My recommendation would be that IPPM does not start work in 
|> |> developing objective video quality estimation algorithms, as this 
|> |> duplicates work already underway in a number of other 
|> groups and IPPM 
|> |> may not have the expertize in this area.
|> |> 
|> |> IPPM may note however that there are some "holes" in the 
|> IPPM metrics 
|> |> as they apply to voice and video over IP.  Packet loss 
|> often occurs 
|> |> in sparse bursts, for example periods of 5-30% loss that 
|> can last for 
|> |> periods of up to several seconds, and IPPM don't have a metric to 
|> |> describe these.
|> |> 
|> |> 
|> |> Regards
|> |> 
|> |> Alan Clark
|> |> Telchemy
|> |> 
|> |>  
|> |> 
|> |> -----Original Message-----
|> |> From: Geib, Ruediger [mailto:[email-masked]]
|> |> Sent: Tuesday, February 27, 2007 3:17 AM
|> |> To: [email-masked]
|> |> Cc: [email-masked]
|> |> Subject: RE: [ippm] Application loss metrics
|> |> 
|> |> Dear Nagarjuna
|> |> 
|> |> your proposal to work on application level quality metrics is 
|> |> allright with me. To the best of my knowledge, up to now 
|> there's no 
|> |> objective IP TV/video conferencing quality measurement based on 
|> |> subjective judgements which would correspond e.g. to those models 
|> |> available for telephony services.
|> |> 
|> |> Considering bare "frame loss" however may not result in a 
|> |> satisfactory judgement of the influence of loss on a streaming 
|> |> application. It does not
|> |> characterise:
|> |> 
|> |> - the importance of the frame which was lost within a
|> |>   Group of Pictures. I frames carry full pictures and
|> |>   serve as a reference, P frames serve as a reference,
|> |>   whereas B frames don't serve as a reference to
|> |>   other frames.
|> |> 
|> |> - the impact of the frame drop within a Group of
|> |>   Pictures on the error propagation within a group of
|> |>   pictures. If a part of an I frame is lost, all other
|> |>   frames referrring to the lost section will display
|> |>   errouneous information no matter whether they are
|> |>   received correctly or not. On the other side, if
|> |>   just a B-frame is lost, only a single picture
|> |>   is influenced (and not a complete Group of
|> |>   Pictures).
|> |> 
|> |> - the different size of frames in terms of IP packets
|> |>   per frame. An I frame may be bigger than a P frame. 
|> |>   Both definitely are significantly bigger than
|> |>   B frames.
|> |> 
|> |> For those who aren't familiar with streaming: A Group of 
|> Pictures is 
|> |> the basic way of tranmistting a compressed picture by one 
|complete 
|> |> picture and a set of differential changes to it. Each 
|> frame results 
|> |> in an own picture and the Group of Picture is:
|> |> 
|> |> I - BB - P - BB - P.... until next I. Not all frame types must be 
|> |> present (except I frames).
|> |> 
|> |> Regards,
|> |> 
|> |> Ruediger
|> |> 
|> |> -----Original Message-----
|> |> From: Venna, Nagarjuna [mailto:[email-masked]]
|> |> Sent: Tuesday, February 27, 2007 4:43 AM
|> |> To: [email-masked]
|> |> Subject: [ippm] Application loss metrics
|> |> 
|> |> 
|> |> 
|> |> 
|> |> I have posted an individual draft
|> |> (http://www.ietf.org/internet-drafts/draft-venna-ippm-app-loss
|> |> -metrics-00.tx
|> |> t) that describes the application level metrics. These are 
|> the same 
|> |> metrics I have talked about in the San Diego meeting.
|> |> 
|> |> Comments and feedback are appreciated.
|> |> 
|> |> Thanks,
|> |> Nagarjuna
|> |> 
|> |> 
|> |> 
|> |> --------------------------------------------------------------
|> |> --------------
|> |> --------------------------------------------------
|> |> Service Quality Matters. Test the performance and quality of your 
|> |> VoIP or IP video service at:
|> |> http://www.TestYourVoIP.com
|> |> http://www.TestYourIPVideo.com
|> |> 
|> |> ---------------------------------------------------
|> |> 
|> |> Rüdiger Geib
|> |> 
|> |> T-Systems Enterprise Services GmbH
|> |> Systems Integration
|> |> Rüdiger Geib
|> |> Line of Business Networks & Processes Senior Consultant 
|> |> Deutsche-Telekom-Allee 7, 64295 Darmstadt
|> |> +49 6151 937-2138 (Tel.)
|> |> http://www.t-systems.com
|> |> 
|> |> T-Systems Enterprise Services GmbH
|> |> Aufsichtsrat: René Obermann (Vorsitzender) Executive
|> |> Committee: Lothar Pauly (Vorsitzender)*, Helmut Binder, 
|> Albert Henn, 
|> |> Olaf Heyden*, Katrin Horstmann, Ulrich Kemp, Axel Knobe*, 
|Wilfried 
|> |> Peters*, Dr. Herbert Schaaff*, Zvezdana
|> |> Seeger*
|> |> Handelsregister: Amtsgericht Frankfurt am Main HRB 55933 Sitz der
|> |> Gesellschaft: Frankfurt am Main WEEE-Reg.-Nr. DE87523644 
|> |> *Geschäftsführer gem. § 35 GmbHG
|> |> 
|> |> Notice: This transmittal and/or attachments may be privileged or 
|> |> confidential. If you are not the intended recipient, you 
|> are hereby 
|> |> notified that you have received this transmittal in error; any 
|> |> review, dissemination, or copying is strictly prohibited. If you 
|> |> received this transmittal in error, please notify us 
|> immediately by 
|> |> reply and immediately delete this message and all its 
|attachments. 
|> |> Thank you.
|> |> 
|> |>  
|> |> 
|> |> _______________________________________________
|> |> ippm mailing list
|> |> [email-masked]
|> |> https://www1.ietf.org/mailman/listinfo/ippm
|> |> 
|> |> 
|> |> _______________________________________________
|> |> ippm mailing list
|> |> [email-masked]
|> |> https://www1.ietf.org/mailman/listinfo/ippm
|> |> 
|> |
|> |_______________________________________________
|> |ippm mailing list
|> [email-masked]
|> |https://www1.ietf.org/mailman/listinfo/ippm
|> |
|> 
|