Subject: [ippm] draft-duffield-ippm-burst-loss-metrics-01

From: Yaakov Stein <[email-masked]>
Date: Sun, 1 Nov 2009 16:35:51 +0200

I read the aforementioned draft with interest, as I believe that it is important to have a way of quantifying the burstiness of packet loss.
I must admit to having read it too quickly, as I am traveling and wanted to get some comments in before the upcoming meeting.

While I believe that the metrics defined are a good first step, I must admit to feeling unsatisfied with them.
I simply am not sure that the concept of burstiness has been captured sufficiently well.

Yes, the duration of a packet burst and the frequency of bursts are nice to have,
but first we need to quantify how bursty the loss is - if it is not bursty then these two metrics are meaningless.
For this purpose the draft proposes loss-pair-counts and bi-packet-loss-ratio,
but loss-pair-counts are too raw and bi-packet-loss-ratio doesn't seem to describe the right thing.

Another minor gripe I have with the methodology is the introduction of time.
I think of burstiness in terms of packet (transmit) sequence number.
Is it really necessary to introduce a time scale ? It seems to complicate things.

What did I expect as a metric ? Well, I have become used to the use of Gilbert-Elliott models,
where there are two states - loss and non-loss, with easily measured probabilities of transitions
between the states, and probabilities of loss or not in either state. These probabilities seem to
capture well the subjective idea of burstiness, but it takes a while to get used to them.

Intuitively, after defining the probability of loss, we need to describe the higher moments or cumulants.
So a simple derived metric would be the probability of consecutive packets being lost divided by the PLR^2,
the probability of three consecutive packets lost divided by PLR^3, etc.
As a complement one could do the probability of two randomly chosen distant packets both being lost
divided by PLR^2, etc.

Can someone map the metrics described in the draft to these ideas ?

Y(J)S

---

From: "DUFFIELD, NICHOLAS G (NICK)" <[email-masked]>
Date: Tue, 3 Nov 2009 16:41:00 -0500

Yaakov,

> -----Original Message-----
> From: [email-masked] [mailto:[email-masked]] On Behalf 
> Of Yaakov Stein
> Sent: Sunday, November 01, 2009 9:36 AM
> To: IETF IPPM WG
> Subject: [ippm] draft-duffield-ippm-burst-loss-metrics-01
> 
> I read the aforementioned draft with interest, as I believe that it is 
> important to have a way of quantifying the burstiness of packet loss.
> 
> I must admit to having read it too quickly, as I am traveling and 
> wanted to get some comments in before the upcoming meeting.

Thanks for finding the time to review the draft before the upcoming IETF meeting. Please see responses inline below:

> but first we need to quantify how bursty the loss is - if it is not 
> bursty then these two metrics are meaningless.

We are not trying to reproduce some other definitions of burstiness, e.g., those based on moments (or cumulants), such as the index of dispersion of counts. 

The metrics of duration and frequency of loss episodes directly characterize the (loss) experience of traffic flows, and can be related to application requirements and SLAs. They are transparent, easy to implement, do not depend on a reference model, and are meaningful whether or not the underlying loss patterns are bursty. 

(As an aside, one could formulate a test for the presence or absence of burstiness based on the metric values, according to whether they are together consistent with Poissonian loss).

> 
> For this purpose the draft proposes loss-pair-counts and 
> bi-packet-loss- ratio,

The "end products" of this draft are the metrics in Section 6:

1. Type-P-One-way-Bi-Packet-Loss-Geometric-Stream-Ratio (the packet loss rate) 

2. Type-P-One-way-Bi-Packet-Loss-Geometric-Stream-Episode-Duration (the loss episode duration) 

3. Type-P-One-way-Bi-Packet-Loss-Geometric-Stream-Episode-Frequency (the frequency at which loss episode start)

> 
> but loss-pair-counts are too raw
> 

For exposition, we organized some of the intermediate quantities as what we termed proto-metrics in Section 5, in order to avoid repeated definitions in the end product metrics of Section 6. But these could be rolled up together in implementations. We can add a statement that the proto-metrics themselves need not be reported.

> and bi-packet-loss-ratio doesn't seem to
> describe the right thing.

Bi-packet-loss-ratio is the average loss as reported using the metric methodology. Although it is not the central new metric of this effort (it doesn't inform on burstiness) we believe it is still useful to have the ability to report average loss in the burst loss measurement framework, since it makes the burst-loss metrics standalone for the measurement of average loss.

> Is it really necessary to introduce a time scale ? It seems to complicate

If people want to know the temporal duration and frequency of loss episodes, then time is useful.

> probabilities of transitions

As mentioned above, the metrics here are model independent. On the other hand, the metric values that would be obtained if applied to a Gilbert-Elliot loss process can be related to the parameters of the model. We can detail this in the draft if that would be of interest.

> being lost divided by the PLR^2,

The metrics in the draft in fact work directly with the probabilities of the loss or transmission for two consecutive packets. In particular, the loss-pair-counts, to which you earlier referred, are, when divided by the number of probe pairs, the measured probabilities of these pair events.

> the probability of three consecutive packets lost divided by PLR^3, etc.

In principle, higher order moments could be used to obtain a more detailed statistical picture of the burstiness. However, higher order moments are subject to greater statistical estimation error, and are in any case not needed to answer the basic question as we see it: how frequent and how long are loss episodes? This last point also applies to the "distant packets" matter following.

> Can someone map the metrics described in the draft to these ideas ?

In summary, we can detail the connection to specific models in future versions on the draft.

> Y(J)S

Nick

---

From: <[email-masked]>
Date: Wed, 4 Nov 2009 08:40:37 +0100

Nick, 

[snip]
 
[Yaakov]> What did I expect as a metric ? Well, I have become used to the use of
> Gilbert-Elliott models,

[Nick]: As mentioned above, the metrics here are model independent. On the other hand, the metric values that would be obtained if applied to a Gilbert-Elliot loss process can be related to the parameters of the model. We can detail this in the draft if that would be of interest.

[Ruediger]: Whether related to Gilbert Elliot or modelling in general, a more informative statement on applicability of the metrics is indeed useful. 

Regards,

Ruediger

Deutsche Telekom Netzproduktion GmbH 
Zentrum Technik Einführung 
Technik Internet Backbone, TE142-19
Rüdiger Geib
Heinrich Hertz Str. 3-7
64297 Darmstadt
Tel.: 06151/6282747
Fax: 0251/7985109

Deutsche Telekom Netzproduktion GmbH 
Aufsichtsrat: Dr. Steffen Roehn (Vorsitzender) 
Geschäftsführung: Dr. Bruno Jacobfeuerborn (Vorsitzender), Albert Matheis, Klaus Peren 
Handelsregister: Amtsgericht Bonn HRB 14190 
Sitz der Gesellschaft: Bonn 
USt-IdNr.: DE 814645262

---

From: Vinayak Hegde <[email-masked]>
Date: Thu, 5 Nov 2009 18:17:53 +0530

> important to have a way of quantifying the burstiness of packet loss.
[Snip]

I agree to what Yaakov has said with regards to time. When I think for
burstiness, I would like to have a metric which can be scaled to
multiple packets in succession, so while this draft might start with a
pair of packets, it might be inadequate. The reason being, I would
like to have a better sample with higher resolution of measurement.
Quantification of bursty loss will be more accurate in terms of
percentage when more packets are involved in succession. Also the
order of loss of individual packets in a stream (a bunch of packets
used for measurement) might not be important, what would matter is the
aggregate sample such as there was a loss of 1 packet on average when
10 bursts of 10 packets were used as compared to a loss of 1 loss-pair
per 10 loss-pair packets.

The only counterexample that I can think of is when some packets are
more important that others and there is a periodic 'SYNC' packet send
in a stream. The loss of the SYNC packet would be higher as compared
to other packets if it is a periodic loss every X interval.

Additional comments:

In section 4., a geometric distribution is mentioned. It is not clear
to me why a geometric distribution might be used between successive
time (exponential backoff/damping oscillations). any precedent for
this ?

In section 4.3 / 4.4 What are the use cases for m <= n Why would m be
<= n is not immediately clear.

In section 4.7 Should there be a reference to asymmetric losses ? For
example in a video conferencing app with two participant there is loss
in one direction but not the other leading to loss in video/audio
quality.

In section 5.1 Why is there a differentiation between (1,0) and (0,1)
packets. In burstiness, shouldn't the aggregate statistics matter  and
not the order of loss ?

Finally the selection function is mentioned in several places in the
draft. A couple of examples of each with real-life scenarios could be
useful.

-- Vinayak

---

From: "DUFFIELD, NICHOLAS G (NICK)" <[email-masked]>
Date: Thu, 5 Nov 2009 11:12:28 -0500

Vinayak,

Thanks for reading the draft. Responses are inline below:

> Of Vinayak Hegde
> Sent: Thursday, November 05, 2009 7:48 AM
> To: Yaakov Stein
> Cc: IETF IPPM WG
> Subject: Re: [ippm] draft-duffield-ippm-burst-loss-metrics-01
> 
> > I read the aforementioned draft with interest, as I believe that it 
> > is important to have a way of quantifying the burstiness of packet loss.
> [Snip]
> 

I think we draft authors may have created some unmet expectations as to what this draft is trying to achieve, by too liberal use of the term "burstiness". A more accurate idea of the work would be conveyed by saying it defines "loss episode metrics", specifically, the average length of loss episodes, and the average number of loss episodes per unit time.

The focus on time is because we want to relate the durations of episodes to the temporal requirements of applications. Metrics scaled by packet count rather than time are not so useful because the packet rates of probe and application traffic are in general different.

A great advantage of using average loss episode metrics (duration and frequency) is that they do not require measurement of complete loss bursts (nor loss patterns as in RFC 3357). They only require measurement of the frequencies of the 4 possible outcomes from two successive packets, and these can be estimated from sampling. 

If there is WG interest is trying to formulate metrics which capture general multipacket statistical detail of loss episodes, this could be considered as a different WG item. This would be necessary for metrics of loss episodes other than the average properties (e.g. variance of episode duration). However, that is outside the scope of this draft, which is the simplest increment from RFC 2680 that provides metrics that inform of properties loss episodes, instead of just an average packet loss rate. 

> time (exponential backoff/damping oscillations). any precedent for 

The probes (= packet pairs) are sampling the bi-packet outcomes. So you can think of the geometric distribution as giving rise to (discrete) Poisson probing at the packet pair level, analogous to the single packet Poisson probing for one-way loss measurement in RFC 2680. Geometric sampling in general is described in RFC 2330.

> <= n is not immediately clear.

Consider a set of evenly spaced times at which probe pairs could be sent. From these times and (random) subset of size m <= n is selected, e.g., by means of the geometric sampling just discussed.

> in one direction but not the other leading to loss in video/audio 

The metrics in the draft are one way metrics.

> packets. In burstiness, shouldn't the aggregate statistics matter  and 

In estimating the frequencies of episodes from the samples, it is more accurate to count both the "episode start" samples (0,1) and the "episode end" samples (1,0).

> draft. A couple of examples of each with real-life scenarios could be 

OK; we will elaborate in next version.

Nick

> _______________________________________________
> ippm mailing list
> [email-masked]
> https://www.ietf.org/mailman/listinfo/ippm

---

From: "Murtaza Chiba (mchiba)" <[email-masked]>
Date: Wed, 11 Nov 2009 11:14:23 -0800

I took a look at the draft and it was hard to understand the concepts
without diagrams. Also, I am not sure about the reasons to use geometric
v/s the non-geometric form. Lastly, I could not understand the formulae
very well, are these based on standard probability functions, do I need
to read some other document?

I agree with a previous reviewer that the concepts need more examples to
make it understandable.

Regards,
-murtaza