Subject: [ippm] Application loss metrics

From: "Venna, Nagarjuna" <[email-masked]>
Date: Mon, 26 Feb 2007 22:43:02 -0500

I have posted an individual draft (http://www.ietf.org/internet-drafts/draft-venna-ippm-app-loss-metrics-00txt) that describes the application level metrics. These are the same metrics I have talked about in the San Diego meeting.

Comments and feedback are appreciated.

Thanks,
Nagarjuna

-----------------------------------------------------------------------------------------------------------------------------------------------
Service Quality Matters. Test the performance and quality of your VoIP or IP video service at: 
http://www.TestYourVoIP.com
http://www.TestYourIPVideo.com

_______________________________________________
ippm mailing list
[email-masked] 
https://www1.ietf.org/mailman/listinfo/ippm

---

From: "Geib, Ruediger" <[email-masked]>
Date: Tue, 27 Feb 2007 09:17:29 +0100

Dear Nagarjuna

your proposal to work on application level quality metrics
is allright with me. To the best of my knowledge, up to 
now there's no objective IP TV/video conferencing quality 
measurement based on subjective judgements which would 
correspond e.g. to those models available for telephony 
services.

Considering bare "frame loss" however may not result in a 
satisfactory judgement of the influence of loss on 
a streaming application. It does not characterise:

- the importance of the frame which was lost within a 
  Group of Pictures. I frames carry full pictures and 
  serve as a reference, P frames serve as a reference, 
  whereas B frames don't serve as a reference to 
  other frames.

- the impact of the frame drop within a Group of 
  Pictures on the error propagation within a group of 
  pictures. If a part of an I frame is lost, all other 
  frames referrring to the lost section will display 
  errouneous information no matter whether they are 
  received correctly or not. On the other side, if 
  just a B-frame is lost, only a single picture 
  is influenced (and not a complete Group of 
  Pictures).

- the different size of frames in terms of IP packets 
  per frame. An I frame may be bigger than a P frame. 
  Both definitely are significantly bigger than 
  B frames.

For those who aren't familiar with streaming: A Group 
of Pictures is the basic way of tranmistting a compressed 
picture by one complete picture and a set of 
differential changes to it. Each frame results in an own 
picture and the Group of Picture is:

I - BB - P - BB - P.... until next I. Not all frame 
types must be present (except I frames).

Regards,

Ruediger

-----Original Message-----
From: Venna, Nagarjuna [mailto:[email-masked]]
Sent: Tuesday, February 27, 2007 4:43 AM
To: [email-masked]
Subject: [ippm] Application loss metrics

I have posted an individual draft (http://www.ietf.org/internet-drafts/draft-venna-ippm-app-loss-metrics-00.txt) that describes the application level metrics. These are the same metrics I have talked about in the San Diego meeting.

Thanks,
Nagarjuna 

------------------------------------------------------------------------------------------------------------------------------
Service Quality Matters. Test the performance and quality of your VoIP or IP video service at:
http://www.TestYourVoIP.com
http://www.TestYourIPVideo.com

---------------------------------------------------

Rüdiger Geib

T-Systems Enterprise Services GmbH
Systems Integration
Rüdiger Geib
Line of Business Networks & Processes
Senior Consultant
Deutsche-Telekom-Allee 7, 64295 Darmstadt
+49 6151 937-2138 (Tel.)
http://www.t-systems.com

T-Systems Enterprise Services GmbH
Aufsichtsrat: René Obermann (Vorsitzender)
Executive Committee: Lothar Pauly (Vorsitzender)*, Helmut Binder, Albert Henn, Olaf Heyden*, Katrin Horstmann, Ulrich Kemp, Axel Knobe*,
Wilfried Peters*, Dr. Herbert Schaaff*, Zvezdana Seeger*
Handelsregister: Amtsgericht Frankfurt am Main HRB 55933
Sitz der Gesellschaft: Frankfurt am Main
WEEE-Reg.-Nr. DE87523644
*Geschäftsführer gem. § 35 GmbHG

Notice: This transmittal and/or attachments may be privileged or confidential. If you are not the intended recipient, you are hereby notified that you have received this transmittal in error; any review, dissemination, or copying is strictly prohibited. If you received this transmittal in error, please notify us immediately by reply and immediately delete this message and all its attachments. Thank you.

---

From: "Alan Clark" <[email-masked]>
Date: Tue, 27 Feb 2007 08:19:33 -0500

Ruediger

There is considerable work within the standards community in the development
of objective video measurement techniques, much of which is fairly advanced.
Some of the lead groups include ITU-T Study Group 9 and 12, the ITU-T IPTV
Focus Group, the Video Quality Experts Group (VQEG), ATIS, DSL Forum, TM
Forum.........  
Within IETF, AVT have a working group draft -
draft-ietf-avt-rtcpxr-video-00.txt - which has been under development for
the last year, this focusses more on the transport of metrics rather than
defining new video objective techniques.

Objective video quality estimation algorithms are not east to test
(objectively) and hence the development of test programs and the work of
comparison with subjective test data often becomes complex.  This is one
reason that VQEG exists - it provides a testing function that works with
many other standards organizations.   

With regard to your specific observations on I/B/P frames - this is
absolutely considered within the work done by the various ITU groups and
ATIS.  One technique that has been defined, discussed across multiple
standards organizations and is currently being assessed is based on PSNR
estimation - which is taking into account loss propagation effects through
the GoP as well as other coding parameters.  

My recommendation would be that IPPM does not start work in developing
objective video quality estimation algorithms, as this duplicates work
already underway in a number of other groups and IPPM may not have the
expertize in this area.

IPPM may note however that there are some "holes" in the IPPM metrics as
they apply to voice and video over IP.  Packet loss often occurs in sparse
bursts, for example periods of 5-30% loss that can last for periods of up to
several seconds, and IPPM don't have a metric to describe these.

Regards

Alan Clark
Telchemy

-----Original Message-----
From: Geib, Ruediger [mailto:[email-masked]] 
Sent: Tuesday, February 27, 2007 3:17 AM
To: [email-masked]
Cc: [email-masked]
Subject: RE: [ippm] Application loss metrics

Dear Nagarjuna

your proposal to work on application level quality metrics is allright with
me. To the best of my knowledge, up to now there's no objective IP TV/video
conferencing quality measurement based on subjective judgements which would
correspond e.g. to those models available for telephony services.

Considering bare "frame loss" however may not result in a satisfactory
judgement of the influence of loss on a streaming application. It does not
characterise:

For those who aren't familiar with streaming: A Group of Pictures is the
basic way of tranmistting a compressed picture by one complete picture and a
set of differential changes to it. Each frame results in an own picture and
the Group of Picture is:

I - BB - P - BB - P.... until next I. Not all frame types must be present
(except I frames).

Regards,

Ruediger

-----Original Message-----
From: Venna, Nagarjuna [mailto:[email-masked]]
Sent: Tuesday, February 27, 2007 4:43 AM
To: [email-masked]
Subject: [ippm] Application loss metrics

I have posted an individual draft
(http://www.ietf.org/internet-drafts/draft-venna-ippm-app-loss-metrics-00.tx
t) that describes the application level metrics. These are the same metrics
I have talked about in the San Diego meeting.

Thanks,
Nagarjuna 

----------------------------------------------------------------------------
--------------------------------------------------
Service Quality Matters. Test the performance and quality of your VoIP or IP
video service at:
http://www.TestYourVoIP.com
http://www.TestYourIPVideo.com

Rüdiger Geib

T-Systems Enterprise Services GmbH
Aufsichtsrat: René Obermann (Vorsitzender) Executive Committee: Lothar Pauly
(Vorsitzender)*, Helmut Binder, Albert Henn, Olaf Heyden*, Katrin Horstmann,
Ulrich Kemp, Axel Knobe*, Wilfried Peters*, Dr. Herbert Schaaff*, Zvezdana
Seeger*
Handelsregister: Amtsgericht Frankfurt am Main HRB 55933 Sitz der
Gesellschaft: Frankfurt am Main WEEE-Reg.-Nr. DE87523644 *Geschäftsführer
gem. § 35 GmbHG

Notice: This transmittal and/or attachments may be privileged or
confidential. If you are not the intended recipient, you are hereby notified
that you have received this transmittal in error; any review, dissemination,
or copying is strictly prohibited. If you received this transmittal in
error, please notify us immediately by reply and immediately delete this
message and all its attachments. Thank you.

---

From: "Venna, Nagarjuna" <[email-masked]>
Date: Tue, 27 Feb 2007 10:04:26 -0500

Hello Ruediger,

Thanks for your feedback. The frame classification you have described applies to computing the general 'errored seconds' metric for a specific application. Such text can certainly go into an example section. As Alan mentioned various groups are developing algorithms for video quality estimation and it makes sense to leverage that work in generating some of the metrics described in this draft.

Regards,
Nagarjuna 

http://www.TestYourIPVideo.com-----Original Message-----
To: Venna, Nagarjuna

Dear Nagarjuna

your proposal to work on application level quality metrics is allright with me. To the best of my knowledge, up to now there's no objective IP TV/video conferencing quality measurement based on subjective judgements which would correspond e.g. to those models available for telephony services.

Considering bare "frame loss" however may not result in a satisfactory judgement of the influence of loss on a streaming application. It does not characterise:

For those who aren't familiar with streaming: A Group of Pictures is the basic way of tranmistting a compressed picture by one complete picture and a set of differential changes to it. Each frame results in an own picture and the Group of Picture is:

I - BB - P - BB - P.... until next I. Not all frame types must be present (except I frames).

Regards,

Ruediger

-----Original Message-----
From: Venna, Nagarjuna [mailto:[email-masked]]
Sent: Tuesday, February 27, 2007 4:43 AM
To: [email-masked]
Subject: [ippm] Application loss metrics

Thanks,
Nagarjuna 

Rüdiger Geib

T-Systems Enterprise Services GmbH
Aufsichtsrat: René Obermann (Vorsitzender) Executive Committee: Lothar Pauly (Vorsitzender)*, Helmut Binder, Albert Henn, Olaf Heyden*, Katrin Horstmann, Ulrich Kemp, Axel Knobe*, Wilfried Peters*, Dr. Herbert Schaaff*, Zvezdana Seeger*
Handelsregister: Amtsgericht Frankfurt am Main HRB 55933 Sitz der Gesellschaft: Frankfurt am Main WEEE-Reg.-Nr. DE87523644 *Geschäftsführer gem. § 35 GmbHG

---

From: "Alan Clark" <[email-masked]>
Date: Tue, 27 Feb 2007 13:55:59 -0500

Hi Nagarjuna

Actually - some of the groups have already defined an errored seconds-like
metric.  

For voice 
  - the AVT's RTCP HR draft incorporates a "concealed seconds" metric 
  - the burst metrics in RFC3611 equate to severely errored periods.

For video - ATIS are using - "degraded seconds" and "severely degraded
seconds"

For voice and video - TM Forum uses - "degraded service quality event"

Some of these metrics can be related together - for example you can derive
"severely *** seconds" from sparse burst/gap metrics.

As a matter of purely historical interest - the term errored second comes
from PCM and relates to bit errors.  If you compute the voice quality
resulting from a typical PCM-era "severely errored" bit error rate this
equates closely with the voice quality during burst periods as defined in
RFC3611.

Regards

Alan

-----Original Message-----
From: Venna, Nagarjuna [mailto:[email-masked]] 
Sent: Tuesday, February 27, 2007 10:04 AM
To: Geib, Ruediger
Cc: [email-masked]
Subject: RE: [ippm] Application loss metrics

Hello Ruediger,

Thanks for your feedback. The frame classification you have described
applies to computing the general 'errored seconds' metric for a specific
application. Such text can certainly go into an example section. As Alan
mentioned various groups are developing algorithms for video quality
estimation and it makes sense to leverage that work in generating some of
the metrics described in this draft.

Regards,
Nagarjuna 

-------------------------------------------------------------------

Dear Nagarjuna

Regards,

Ruediger

-----Original Message-----
From: Venna, Nagarjuna [mailto:[email-masked]]
Sent: Tuesday, February 27, 2007 4:43 AM
To: [email-masked]
Subject: [ippm] Application loss metrics

I have posted an individual draft
(http://www.ietf.org/internet-drafts/draft-venna-ippm-app-loss-metrics-00txt
) that describes the application level metrics. These are the same metrics I
have talked about in the San Diego meeting.

Thanks,
Nagarjuna 

Rüdiger Geib