Subject: RE: [ippm] metric requirements

From: "Dvorak, Charles A (Chuck), ALASO" <[email-masked]>
Date: Thu, 3 Oct 2002 18:17:10 -0400

Further comments on packet loss:

Recommendation Y.1541 of the ITU-T defines 6 IP layer QoS classes, NI-to-NI. All classes except best effort have a 0.1% packet loss objective, measured over an interval of a minute. One reason for this objective is an acknowledgment that in real networks, packets occur in bursts. So the "several percent packet loss" tolerance that many people have stated is really only for the unrealistic, uniformly distributed random case. When this 0.1% objective is met, speech quality will almost always be acceptable--as long as they don't all occur in just one or two bursts.

Just FYI, AT&T Labs proposed a new approach (not adopted yet) that took a three-pronged approach to setting objectives for loss packets: One objective keeps the quality as good as a PSTN call; the second objective allows a few bursts spread out causing some, but not serious, degradation; the third objective limits the maximum sequential loss, so as to not have any bursts that in and of themselves cause an unacceptably annoying impairment during the minute interval.

Finally, the E-model of G.107 was recently modified to accommodate (albeit crudely) the effects on speech quality of bursty packet loss, which should dispel the notion that the E-model is not for VoIP use. It is!

Anyway a key message here is that (supporting Alan's message) during high burtsy losses, one goes from very good speech quality to unusable moments. Unlike random losses, bursts make you fall off a cliff.

Chuck Dvorak
AT&T Labs

-----Original Message-----
From: Alan Clark [mailto:[email-masked]]
Sent: Monday, September 30, 2002 3:19 PM
To: 'stanislav shalunov'; [email-masked]
Cc: [email-masked]
Subject: RE: [ippm] metric requirements

Fortunately many codecs behave in a consistent way and don't generate triple
the amount of traffic in order to improve resilience!!

Work done within ITU SG12, ETSI TIPHON and a few other groups has
characterized the behavior of a wide range of codecs under packet loss
conditions up to 20-30%.  It can however be a little tricky deducing if
packet loss concealment is in use from the info available in RTP headers!

One key point when considering the effects of packet loss on VoIP is the
distribution of packet loss.  This does not behave, as many would suggest,
as a simple burst channel exhibiting only consecutive loss.  Bursts of loss
are often sparse (e.g. 20%) and extend for periods of seconds - largely
caused by periods of congestion.  You may also see the extreme case of
bursts of consecutive loss extending for hundreds of packets, due to link
failures.
The effect of this is that call quality will be fine for some period of
time, and then experience several seconds of badly degraded quality before
improving again.

Alan Clark

-----Original Message-----
From: [email-masked] [mailto:[email-masked]]On Behalf
Of stanislav shalunov
Sent: Monday, September 30, 2002 12:10 PM
To: [email-masked]
Cc: [email-masked]
Subject: Re: [ippm] metric requirements

"H.E.Holland" <[email-masked]> writes:

> Where might I find quantitative metric requirements for services
> such as VoIP? E.g. what are recommended values for one-way delay and
> packet loss for a VoIP service?

While others pointed out the ITU documents for one-way delay, I don't
think we answered the `loss' part of the question.

There can possibly be no answer, because codecs have various degrees
of tolerance and redundancy.  Relevant gedanken Experiment: Imagine
that your favorite codec were to send all its packets in triplicate;
that would change the threshold of tolerance for loss, wouldn't it?

--
Stanislav Shalunov		http://www.internet2.edu/~shalunov/

"I didn't attend the funeral, but I sent a nice letter saying that I
approved of it."				-- Mark Twain
_______________________________________________
ippm mailing list
[email-masked]
http://mailhost.advanced.org/mailman/listinfo/ippm

_______________________________________________
ippm mailing list
[email-masked]
http://mailhost.advanced.org/mailman/listinfo/ippm

_______________________________________________
ippm mailing list
[email-masked]
http://mailhost.advanced.org/mailman/listinfo/ippm

---

From: "Telchemy Standards Tracking" <[email-masked]>
Date: Thu, 3 Oct 2002 21:26:54 -0400

This may be getting into too much detail (however is my favorite subject!!!)

We recently compared the burst parameters of multiple large traces using
both the consecutive loss and a Gilbert model.  In addition to analyzing
pure packet loss we also looked at the distribution of discards due to
jitter.  The result showed clearly that bursts of both loss and loss/discard
were typically 20-200 packets in length with loss densities of 20-30%,
whereas the consecutive loss model would have suggested bursts averaging 1.5
packets!!!  This is an easy experiment to duplicate and can be performed
with Sue Moon or Wenyu Jiang's trace files. For those able to make
measurements on core IP networks then it is quite possible that loss would
primarily occur to to route changes or link failures and hence that loss may
often be consecutive, this should not however be used to infer that loss is
always consecutive.

I agree with Chuck that a multi-pronged approach is essential however I
would suggest that a Gilbert/ Gilbert-Elliott or 3/4-state Markov model
based approach would be preferable to a consecutive loss model.  We use a
4-state Markov model which combines the ability of Gilbert-Elliott to
capture long sparse bursts with the ability of a 2-state model to capture
consecutive loss, this is computationally efficient and produces all the
statistics needed.

The E Model modifications that were made to support bursty loss
unfortunately focus only on consecutive loss - this is a definite
improvement however still does not reflect conditions that appear quite
common on some networks (including both Enterprise IP networks and the
Internet).  If the E model is used for planning (it's original purpose) then
you don't know what the distribution of packet loss is - hence the current E
model provides a good approach.  In a monitoring (or simulated) enviroment
then it may be possible to see what actually happened in which case a more
sophisticated approach may be justified.

Regards

Alan Clark
Telchemy

-----Original Message-----
From: [email-masked] [mailto:[email-masked]]On Behalf
Of Dvorak, Charles A (Chuck), ALASO
Sent: Thursday, October 03, 2002 6:17 PM
To: [email-masked]; stanislav shalunov;
[email-masked]
Cc: [email-masked]; Ash, Gerald R (Jerry), ALASO; Tarapore, Percy S,
ALASO; [email-masked]; [email-masked]
Subject: RE: [ippm] metric requirements

Recommendation Y.1541 of the ITU-T defines 6 IP layer QoS classes, NI-to-NI.
All classes except best effort have a 0.1% packet loss objective, measured
over an interval of a minute. One reason for this objective is an
acknowledgment that in real networks, packets occur in bursts. So the
"several percent packet loss" tolerance that many people have stated is
really only for the unrealistic, uniformly distributed random case. When
this 0.1% objective is met, speech quality will almost always be
acceptable--as long as they don't all occur in just one or two bursts.

Just FYI, AT&T Labs proposed a new approach (not adopted yet) that took a
three-pronged approach to setting objectives for loss packets: One objective
keeps the quality as good as a PSTN call; the second objective allows a few
bursts spread out causing some, but not serious, degradation; the third
objective limits the maximum sequential loss, so as to not have any bursts
that in and of themselves cause an unacceptably annoying impairment during
the minute interval.

Finally, the E-model of G.107 was recently modified to accommodate (albeit
crudely) the effects on speech quality of bursty packet loss, which should
dispel the notion that the E-model is not for VoIP use. It is!

Anyway a key message here is that (supporting Alan's message) during high
burtsy losses, one goes from very good speech quality to unusable moments.
Unlike random losses, bursts make you fall off a cliff.

Chuck Dvorak
AT&T Labs

-----Original Message-----
From: Alan Clark [mailto:[email-masked]]
Sent: Monday, September 30, 2002 3:19 PM
To: 'stanislav shalunov'; [email-masked]
Cc: [email-masked]
Subject: RE: [ippm] metric requirements

Alan Clark

---

From: Kavé Salamatian <[email-masked]>
Date: Sat, 5 Oct 2002 10:00:11 +0200

I agree fully with the observation made. We have shown before that a Hidden
Markov model with at most 4 states is sufficient for modelling losses in
Internet (Hidden Markov Modeling for network communication channels, M.R
Salamatian, S. Vaton, SIGMETRICS 2001, Cambridge, Massachusett
http://www-rp.lip6.fr/~salamat) even long bursts !

Bests

Kv
-----Message d'origine-----
De : [email-masked] [mailto:[email-masked]]De la part
de Telchemy Standards Tracking
Envoyé : vendredi 4 octobre 2002 03:27
À : Dvorak, Charles A (Chuck), ALASO; stanislav shalunov;
[email-masked]
Cc : [email-masked]; Ash, Gerald R (Jerry), ALASO; Tarapore, Percy S,
ALASO; [email-masked]; [email-masked]
Objet : RE: [ippm] metric requirements

Regards

Alan Clark
Telchemy

-----Original Message-----

Chuck Dvorak
AT&T Labs

-----Original Message-----
From: Alan Clark [mailto:[email-masked]]
Sent: Monday, September 30, 2002 3:19 PM
To: 'stanislav shalunov'; [email-masked]
Cc: [email-masked]
Subject: RE: [ippm] metric requirements

Alan Clark

---

From: "Alan Clark" <[email-masked]>
Date: Sat, 5 Oct 2002 10:17:29 -0400

>Kv

Thanks Kave,

Interestingly enough - much of the the work done in modeling packet loss
distributions duplicates work done in the 1960-1980 time frame on modeling
bit error distributions in telephone channels.  Obviously it is widely known
that the 1960's work of Gilbert and Elliott was focused on this area,
however many others produced models based on the distribution of inter-error
gap lengths (e.g. Mandelbrot) as well as Markov models.
Part of my research work in the early 80's was to model the performance of
adaptive FEC/ARQ protocols over telephone channels - which obviously has
some sensitivity to the distribution of bit errors - and hence I spent a
fair amount of time looking at this issue, seeing to what extent these
models could represent "real" distributions.  If you go back through the
literature you can see many similar observations made for both bit error and
packet loss distributions - although the underlying mechanisms that cause
loss to occur are quite different (i.e. impulse & white noise vs congestion
& link failures).  Is this similarity due to the fact that we tend to
observe the available data through the "eyes" of certain commonly used
models or due to inherent similarities in the properties of the data?

Regards

Alan

---

From: "Telchemy Standards Tracking" <[email-masked]>
Date: Sat, 5 Oct 2002 14:12:27 -0400

[Continuation of previous reply]

> Kv

Kave

The earliest reference I've found to the use of more complex Markov models
to represent bit error distributions is that of Fritchman - IEEE Inf
Theory -13, 1967.

Gap length models can also produce quite reasonable distributions of packet
loss, and have been quite widely used.  I recently looked at a simple
exponential model of gap lengths and managed to get burst length/ burst
density scatter plots that were "similar" to those measured from Internet
trace data using a Gilbert model however the distribution of burst lengths
was quite dissimilar.

An interesting question with regard to the sufficiency of a 4-state model is
over what period is it sufficient?  Losses are due in part to "random"
events such as link failures and in part to LAN, Access link or core network
congestion - which lead to short term correlation and potentially long term
periodicity.

Regards

Alan Clark

---

From: stanislav shalunov <[email-masked]>
Date: 15 Oct 2002 15:29:49 -0400

"Dvorak, Charles A (Chuck), ALASO" <[email-masked]> writes:

> Recommendation Y.1541 of the ITU-T defines 6 IP layer QoS classes,
> NI-to-NI. All classes except best effort have a 0.1% packet loss
> objective, measured over an interval of a minute.

It is fine to define objectives such as this one.  This objective
would be better if loss correlation were considered.

However, loss is very different from delay for the purposes of VoIP.
High latency cannot be compensated for, and therefore, there are
objective thresholds that psychologists can ascertain and that can
further be codified in standards and recommendations; any given loss,
on the other hand, can be compensated for by an appropriate error
correction technique, and therefore, any numbers one comes up with are
simply arbitrary design goals.

It is possible to build codecs that will compensate for 0.1% packet
loss (these codecs will require certain resiliency to packet loss,
hence, they will be redundant to a certain extent).  By changing the
extent of redundancy, one can compensate for 1% packet loss or any
other given number.

This difference (objective law of human perception vs. arbitrary
design goal) between latency and loss is important.

This message is designed to be viewed at 600 dpi.
_______________________________________________
ippm mailing list
[email-masked]
http://mailhost.advanced.org/mailman/listinfo/ippm