Subject: Re: [ippm] draft-ietf-ippm-owdp: optional fields

From: stanislav shalunov <[email-masked]>
Date: 05 Feb 2001 19:40:01 -0500

"Henk Uijterwaal (RIPE-NCC)" <[email-masked]> writes:

> * Experimental error.
> 
>   This follows from RFC2330 which indicates that experimental errors
>   for OWD should be reported.  Reporting errors becomes even more 
>   important when OWD sessions can be set up between different devices
>   operated/manufactured by different organizations.  (Typical example,
>   if one sync's his clock from a nearby NTP server, there will be a large
>   statistical error on the time compared to a situation where one has 
>   a GPS receiver inside the device.  These errors propagate into the
>   result, and should be quoted in order to fully appreciate the 
>   final result for a OWD measurement).
> 
>   This number can be expressed using the same format as an NTP timestamp.

What would this error bar tell us?  The maximum possible error?
Average error?  Median error?  Square root of average error square?
95th (90th, 99th) percentile of error?

Instead of dealing with this ourselves, we decided that it makes sense
to delegate precision definition to time protocol people in a way that
does *not* imply that hosts are running NTP software:

   Sender Precision and Receiver Precision are signed integers in the
   range +32 to -32 indicating the precision of the corresponding
   clocks, in seconds to the nearest power of two, as described in
   RFC 958.

Does this make sense to you?

If it does, then your suggestion is to place this in every packet,
optionally, rather than have it negotiated at setup time as clock
property, right?

> * Clock status.
>   This word should indicate whether the clock is sychronized or not.
>   This is typically necessary during startup and to detect experimental
>   problems.

Synchronized to what in what sense?  It'd be nice to avoid
NTP-specific fields.

>   NTP use 13 bits for this, see /usr/local/include/time.h on your unix
>   box for the definition.

/usr/include/time.h on my BSD machine doesn't have this information.
Could you point me to a URL or specify your Unix flavor?

>   In order to accomodate other time keeping mechanisms, we could use
>   1 word of the send message, and split that into a few bytes to
>   indicate the time keeping method, followed by 13 others if that's
>   NTP or N others for any other timekeeping mechanism.

This suggests to me that the meaning of this field would be up to the
application writers to determine, with only NTP-using implementations
interoperating with each other.  Doesn't look very good, if that's the
case.

Can you please clarify?

> Page 6: "In unautheniticated mode, KID, ..."  
>         but KID is only defined in the next paragraph

"In unauthenticated mode, KID, Token, and Client-IV are unused" was my
way of saying "pay no attention to fields called that on the diagram
if mode is unauthenticated".

Then the text proceeds to explain to to treat them in the next
paragraph in the other two modes.

Should these paragraphs be exchanged?

> Page 6: The response message: I think the text should specify here
>         that YES==0 and NO != 0

The name of the field was changed from "Yes/No" to "Accept" in working
version of the draft.  Wording was changed to:

A zero value in the Accept field means that the server accepts the
authentication and is willing to conduct further transactions.  Any
non-zero value means that the server does not accept the
authentication provided by the client or, for some other reason, is
not willing to conduct further transactions in this OWDP-Control
session.

> Page 8/9, creating sessions:
>         I'm not sure how the sender can say anything about the receiver
>         precision during setup of a session.

I think not:

   Sender Precision is meaningful only if Conf-Sender is not set.
   Receiver Precision is meaningful only if Conf-Receiver is not set.

>         Related to this, precision may vary during a session as
>         clocks tend to drift.

That's (my interpretation) of your first suggestion.

>         Packet length: this does not seem to deal with sessions where
>         one sends packets of varying length.

No, it doesn't.  Shall we insert a way to specify a Turing machine
that will output the required sequence of packet lengths? :-)

I think this, and non-Poisson distributions, lie in the realm of
extension once we have a working protocol.

>         Packets: there should a value for infinite.

This will complicate session results retrieval quite a bit.  At the
moment, you complete the session and get the results (which, in
particular, serves the purposes of non-disturbing test in progress by
control part of the protocol).

If you have sessions with infinite numbers of packets, I'd say you
should no longer be able to retrieve their results.  Is that what you
mean, too?

> Page 11/12: Why do the 1st and 3rd message have a specific value in the
>         first 8 bytes, while the second 1 only has 0 or non-zero. 
>         Wouldn't it be easier to simply use values for the different
>         messages (for example 1=start, 2=ack, 3=stop, 4=... ).  

The specific values are commands, the zero/non-zero values are yes/no
responses.  The intention is to be able to specify later a number of
(non-zero) error codes that would give a clue as to what happened.

It's probably premature to do it now, but hooks for doing it are in
the draft.  (Only add a new section.)

> Page 16: Shouldn't we specify that the sequence numbers increase by 1?

I somehow thought that "sequence number" has this meaning already, but
I guess it wouldn't hurt to say it explicitly.  Thanks!

Does "Sequence numbers start with 0 and are incremented by 1 for each
subsequent packet." sound good?

> Page 18: RFC958 is obsolete and has been replaced by 1305.

Yuck.  Sorry about this.  All references are changed.

> Page 19: " [RIPE] Ripe Test-Traffic home... " should be
>          " [RIPE] RIPE NCC Test-TRaffic Measurements home..."

Corrected.  Thank you.

> Page 19: [RIPE-NLUUG] : The URL should be:
> 
>           http://www.ripe.net/test-traffic/Talks/9805_nluug.ps.gz

Corrected.  Thank you.

-- 
Stanislav Shalunov		http://www.internet2.edu/~shalunov/

"The power of accurate observation is commonly called cynicism by
those who have not got it."			-- G. B. Shaw
_______________________________________________
ippm mailing list
[email-masked]
http://mailhost.advanced.org/mailman/listinfo/ippm

---

From: "Henk Uijterwaal (RIPE-NCC)" <[email-masked]>
Date: Tue, 6 Feb 2001 11:05:51 +0100 (CET)

On 5 Feb 2001, stanislav shalunov wrote:

> What would this error bar tell us?

The standard deviation of the distribution that one would get when taking
a large number of samples from this clock and comparing them to an
(imaginary) absolute time source.  From that, one can always and easily
calculate percentiles (95th is about 2 sigma, 99 is about 3) as well as
any other quantity.

>  The maximum possible error?

This is infinite by definition or perhaps 2^32 seconds in a 32 bit world.

> Does this make sense to you?

RFC958 is obsolete and replaced by 1305.  But besides that, yes, I don't
think that we should assume that people use NTP, but just indicate how
accurate one thinks the clock is.

> optionally, rather than have it negotiated at setup time as clock

Yes, of course, as it can vary from one measurement to another.

A typical scenario when switching on a GPS device is, is that it doesn't
have a clue what the time is and hence has a large error.  When it picks
up a satellite, the error is reduced to something of the order of 0.5
seconds.  When it starts to see more satellites, the error drops
dramatically again when it gets are rough guess of the position, then
continues to improve when it can do a better determination of the position
of the receiver eventually reaching something like 100ns.  The error may
go up again if the receiver loses sight of some of the satellites, then
back down when it picks up another satellite.

These fluctuations can happen in a matter of minutes, so any error
negoatiated at setup is valid for at most 1 minute.

> Synchronized to what in what sense?  It'd be nice to avoid

"Synchronized to an external time-source" or more general "reliable".

What one sees in practice, is that people cut antenna cables from GPS
receivers, filter on NTP packets to a nearby stratum-1, pull the plug from
their DCF77 receivers, etc, etc.  At this point, the measurements become
unreliable, even though the mechanism for doing them might still work.

What I propose is a work to record the clock status, so, after the data
has been collected, one can easily reject measurements where one
suspects problems with the hardware.

The alternative would be to stop the measurements as soon as sender or
receiver suspects a clock problem, but then one cannot use the same data
to measure packet loss continously _and_ one has to go through a setup
cycle for every small hickup.

> Could you point me to a URL or specify your Unix flavor?

Linux, FreeBSD.

> interoperating with each other.  Doesn't look very good, if that's the

No, all I'm saying is N bits for "time keeping mechanism", (32-N) for
the clock status, say, with N=3

  0 1 2 3 4 5 6 7 ...  32
  0 0 1 ....               NTP, with 3..16 the NTP bits
  0 1 0 ...                Read GPS receiver over the bus, with
                           3..8 the number of sats seens
  ...

when analyzing the data, one looks for the first 3 bits, then decides
if the clock reliable.

> Should these paragraphs be exchanged?

I generally don't like abbreviations to appear before their definition.

> extension once we have a working protocol.

OK.

> should no longer be able to retrieve their results.  Is that what you

This is what we do at the moment: sender starts sending, receiver starts
receiving, and once a day we collect what was sent in the last day.  The
receiver software writes the data to a dffirent file each day, but we
don't see why one should stop the measurement for that.

> the draft.  (Only add a new section.)

OK

> subsequent packet." sound good?

Yes.

Henk

------------------------------------------------------------------------------
Henk Uijterwaal                    Email: [email-masked]
RIPE Network Coordination Centre     WWW: http://www.ripe.net/home/henk
Singel 258                         Phone: +31.20.5354414
1016 AB Amsterdam                    Fax: +31.20.5354445
The Netherlands                   Mobile: +31.6.55861746
------------------------------------------------------------------------------

As long as you don't tell your friends how I played the hand,
then I won't tell my friends how you defended it.                 (Anonymous)

---

From: stanislav shalunov <[email-masked]>
Date: 06 Feb 2001 16:06:03 -0500

"Henk Uijterwaal (RIPE-NCC)" <[email-masked]> writes:

> The standard deviation of the distribution that one would get when
> taking a large number of samples from this clock and comparing them
> to an (imaginary) absolute time source.  From that, one can always
> and easily calculate percentiles (95th is about 2 sigma, 99 is about
> 3) as well as any other quantity.

Assuming you have Gaussian distribution of errors with no systematic
error, this is true.  One may argue why externally NTP-synchronized
clocks would have Gaussian distribution of errors in case of symmetric
routing, but in general it doesn't have to be true.

For example, I very much doubt that reading GPS hardware or using
locally GPS-synchronized clock would give you that.  And, of course
you can have unknown systematic errors (e.g., with CDMA time sources).

> This is infinite by definition or perhaps 2^32 seconds in a 32 bit world.

By definition of what?  If you have local GPS hardware that gives you
time with resolution of 1ms, you very much have maximum error.

This is not to say that you won't have standard deviation for whatever
distribution of errors you have.  The points are that 

* no single parameter describes errors adequately (such description,
  even if errors are independent, would require an R->R extended
  function);

* you may not know what your error distribution is (but usually have a
  rough idea about their order of magnitude).

> RFC958 is obsolete and replaced by 1305.

All references replaced per your previous message.

> But besides that, yes, I don't think that we should assume that
> people use NTP, but just indicate how accurate one thinks the clock
> is.

Do you think that what NTP calls precision is sufficient for this
purpose?  That's what we have now (per session, not per packet as you
ask for).

> "Synchronized to an external time-source" or more general "reliable".

This looks like 1 bit of information to me rather than 13 or 32 bits.
Should we think about it as conveying 1 bit?  ("I think I have time
that something external to me, such as a GPS receiver or NTP server,
claims to be UTC.")

> > > /usr/local/include/time.h

> Linux, FreeBSD.

Since FreeBSD uses /usr/local for non-system software, and Linux uses
it for non-packaged software, this is likely some file you have
installed on your systems.  How about 1 bit, though?

> the clock status, say, with N=3

We'd need a rather long list of potential time sources this way,
wouldn't we?  What if it's timeslave?  Local CDMA?  Timed to NTP
server?  GPS over serial with NMEA?  And so forth...

> don't see why one should stop the measurement for that.

Let us think more about implications of infinite irretrievable
sessions for a day or two.

Thanks a lot for all the comments!

"Nuclear war would really set back cable [television]."  -- Ted Turner
_______________________________________________
ippm mailing list
[email-masked]
http://mailhost.advanced.org/mailman/listinfo/ippm

---

From: Stephen Donnelly <[email-masked]>
Date: Wed, 07 Feb 2001 11:01:05 +1300

> time with resolution of 1ms, you very much have maximum error.

Sure, until lightning hits it, or the UPS it's on fails, or there aren't
enough satellites above the horizon, or someone unplugs the patch acble
to it. These things all happen in practice, so there's no such thing as
a maximum error of 1ms, whatever the manufacturer says.

> sessions for a day or two.

Why should it bwe irretreviable? Henk just indicated that it need not
be. Why should data be retrieved only after a measurement 'session' has
been shut down? What if you want some kind of 'near real time'
monitoring/display of delay?

Stephen.
-- 
-----------------------------------------------------------------------
    Stephen Donnelly (BCMS)             email: [email-masked]
    WAND Group                      Room GG.15 phone +64 7 838 4086
    Computer Science Department, University of Waikato, New Zealand
-----------------------------------------------------------------------
_______________________________________________
ippm mailing list
[email-masked]
http://mailhost.advanced.org/mailman/listinfo/ippm

---

From: stanislav shalunov <[email-masked]>
Date: 06 Feb 2001 17:18:41 -0500

Stephen Donnelly <[email-masked]> writes:

> Why should it bwe irretreviable?

What I meant to say, irretrievable by means of OWDP-Control, unless
one wants to allow partial retrivals in a separate session.  This
would naturally mean decreased robustness (how many losses occured
after the last packet?)

> Why should data be retrieved only after a measurement 'session' has
> been shut down?

The motivation to do this would be to be able to ensure complete
transmission of session results, and to distinguish losses due to host
failures from losses due to network conditions.

(I was recording this session which was supposed to have 1000 packets
in it.  Last packet I have seen was packet number 564, with a total of
532.  Were packets 565-999 dropped by the network or did my peer
crash?)

> What if you want some kind of 'near real time' monitoring/display of
> delay?

You'd run short sessions.  (Or, at least, that was the understanding
of how it would work.)

Beware of Programmers who carry screwdrivers.    -- Leonard Brandwein
_______________________________________________
ippm mailing list
[email-masked]
http://mailhost.advanced.org/mailman/listinfo/ippm

---

From: stanislav shalunov <[email-masked]>
Date: 06 Feb 2001 17:44:56 -0500

Stephen Donnelly <[email-masked]> writes:

> to it.

These would all be covered by unsynchronized case.  Presumably,
whatever you use for your error estimate parameter(s) would be
discarded when not synchronized.

> These things all happen in practice, so there's no such thing as

How do you propose the software implementing the protocol would deal
with broken equipment that claims it's synchronized when it's not?
And how having 1 or 1000 parameters describing the errors would change
the situation?

"Nuclear war can ruin your whole compile."          -- Karl Lehenbauer
_______________________________________________
ippm mailing list
[email-masked]
http://mailhost.advanced.org/mailman/listinfo/ippm

---

From: Steven Van den Berghe <[email-masked]>
Date: Wed, 07 Feb 2001 09:45:03 +0100

> You'd run short sessions.  (Or, at least, that was the understanding

Is the overhead of setting up and tearing down sessions not too much a
burdon here? An 'intermediate result retrieval function' would imho not
complicate the protocol too much (the only extra state needed in the
control level would be a pointer to the last result transmitted to the
client) and the request of intermediate results to the control plane
would also detect host faillures.

Steven

--
Steven Van den Berghe
[email-masked]
Workgroup Broadband-Networks
Department Information Technology
Ghent University - Belgium
Phone:  +32 (0)9 267 35 86 | Fax  :  +32 (0)9 267 35 99
*-*-*-*-*-*-*-*-*-*-*-*-*-*-*-*-*-*-*-*-*-*-*-*-*-*-*-*-*
Programming today is a race between software engineers
striving to build bigger and better idiot-proof programs,
and the Universe trying to produce bigger and better
idiots. So far, the Universe is winning. - Rich Cook
*-*-*-*-*-*-*-*-*-*-*-*-*-*-*-*-*-*-*-*-*-*-*-*-*-*-*-*-*

---

From: "Henk Uijterwaal (RIPE-NCC)" <[email-masked]>
Date: Wed, 7 Feb 2001 10:36:40 +0100 (CET)

> 532.  Were packets 565-999 dropped by the network or did my peer

You don't know, but that doesn't mean that you cannot already do something
useful with the 532 packets that you did receive. 

For those 532 points, you can report that packet loss is at least 32/564 =
6% and a distribution of the delays during the time that data came in.
For operational purposes, this is already quite useful even though it is
clearly insufficient data for a full scientific analysis.

> You'd run short sessions.  (Or, at least, that was the understanding

I don't think that this will work due to the overhead being involved.

What I noticed from looking at the data, is that 10 to 20 minutes worth of
data is often enough to see that something changed on the network that
cannot be attributed to a statistical fluke or a user overloading the
line. This can be used for network monitoring systems.

However, I would not want to restart all 50 measurements sessions that I
have running on a node every 10 minutes.  At the moment, setting up all
those 50 sessions takes of the order of a minute, so this would translate
into a dead-time of O(10)%.  But for a measurement device, I typically
think about dead-times of O(0.1)% or less.  I'm sure that some
optimalization can be done, but I doubt that one can gain 2 orders of
magnitude here.

Henk

---

From: "Henk Uijterwaal (RIPE-NCC)" <[email-masked]>
Date: Wed, 7 Feb 2001 11:01:37 +0100 (CET)

> routing, but in general it doesn't have to be true.

Then you should report 2 errors: statistical and systematic. 

> time with resolution of 1ms, you very much have maximum error.

No, this only applies in a specific case: it has seen enough satellites to
determine its position for a while and it had time to stabilize its
internal PLL.  If the same receiver sees only 1 satellite, after just
being switched on, the error can be larger, even if the manufacturers of
the devices claim otherwise.

> purpose?  That's what we have now (per session, not per packet as you

I think the NTP error estimates are sufficient (or at least the best we
have). Per session is insufficient, as the error estimates can change
rapidly.

> server?  GPS over serial with NMEA?  And so forth...

Yes, true.

But, when something goes wrong, I'd like to be able to check what went
wrong.  So how about: 

 bit 0 : 0/1 for clock sync'ed or unsync'ed
 bit 1 : 0/1 for additional data
 bit 2.. : additional data.

For a quick look, look at bit 0. If you want to debug, look at bit 1.  If
the sender included data, then it will be set and you can debug.

Henk

---

From: stanislav shalunov <[email-masked]>
Date: 07 Feb 2001 11:16:06 -0500

Steven Van den Berghe <[email-masked]> writes:

> the request of intermediate results to the control plane would also
> detect host faillures.

Not really, since you'd be talking to one host and another host might
have failed.  (E.g., sender failed, you're retrieving the results from
the receiver; situation becomes even worse if the five logical roles
are spread among five hosts.)

"Which one is worse?  Both are worse."		-- V. I. Lenin
_______________________________________________
ippm mailing list
[email-masked]
http://mailhost.advanced.org/mailman/listinfo/ippm

---

From: stanislav shalunov <[email-masked]>
Date: 07 Feb 2001 11:46:47 -0500

> Then you should report 2 errors: statistical and systematic. 

I don't think it would work; if you know your systematic error the
best way to "report" it would be to subtract it from the timestamps.

> the devices claim otherwise.

The hardware that I am familiar with indicates whether it is
synchronized (knows its position, and, therefore, time).  It might not
say whether it is synced in 2D or 3D mode (with 3 or 4 satellites),
but for static (in space) computers it is not really an issue.

> have). Per session is insufficient, as the error estimates can change

NTP provides your implementation with error estimates.  I assume you
mean the dispersion to the server you are synchronized to.  This
dispersion has some value to it, but it's far from being a real error
estimate.  E.g., when NTP software on Abilene Surveyors indicates
10-50us dispersion, it can typically be off by as much as 200us (and
this is under network conditions that are probably as good as it can
get for NTP).

Further, while NTP at least gives you some numbers (good or bad) many
other mechanisms might not give you that.

I don't understand what you would do in these situations.  Further,
since these error estimates will inevitably be generated using
completely different means you'd end up comparing apples to oranges.

You suggest having one parameter for measuring errors.  What if
somebody came and suggested to put dispersion, median, maximum error,
third momentum, fourth momentum, 50th percentile, and 95th percentile?
In general, these are all independent error characterizations.  We
have to draw the line somewhere.  And it should be drawn in such a way
that it does not favor one specific implementation of time
synchronization (NTP) at the expense of others.

That is why we have (rough) precision rather than some more elaborate
scheme.  Would you be satisfied with this precision information
(-32..32) in each packet, optionally (or mandatory if space permits)?

> the sender included data, then it will be set and you can debug.

Do we assign meaning to the additional bits or is it an
opaque/proprietary field?  If we assign meaning to them, it must not
be NTP-specific.  If we keep an opaque field, it would naturally
hinder inter-operability between different vendors.  ("Hmm, it *says*
it's not synced in the first bit, but this further data tells me time
is actually OK.  Let's trust it.  [...]  Oh wait, I didn't know we're
not talking to the same implementation!")

What did you have in mind?

---

From: "Henk Uijterwaal (RIPE-NCC)" <[email-masked]>
Date: Wed, 7 Feb 2001 18:36:02 +0100 (CET)

> best way to "report" it would be to subtract it from the timestamps.

That is not an error but the calibration of your device.

> mean the dispersion to the server you are synchronized to.  

No, why?  I meant the error estimate on the PLL that NTP gives you, after
it has gone through all its filtering steps and whatever else it does to
extract the correct time.

> I don't understand what you would do in these situations.  

My physics background tells me that I shouldn't do a measurement then.
RFC2330 seems to confirm this.  

What I basically ask is for two things: a time and the best estimate of
the error on that time that the source can give me.   When calculating the
delay, I can then use standard techniques to calculate an error on the
overall result and thus have some indication on how reliable my final
result is.

> Further, since these error estimates will inevitably be generated
> using completely different means you'd end up comparing apples to
> oranges.

This happens all the time and is never a problem.  You ask for a well
defined number and the owner of the clock reports it.

> third momentum, fourth momentum, 50th percentile, and 95th percentile?

I'm asking for one which is reported in about every other paper and from
which the others can be derived, and which is also a de-facto world-wide
standard.

> not talking to the same implementation!")

This cannot happen:

Bit 0 is set based on a time sync'ed/un-sync'ed decision by the sender,
based on the information in the other words, so it is impossible to have a
situation where it says that it is not sync'ed but where the data says
otherwise.

Bit's 1 and further are intended for debugging.  If the source is under my
control, I know which implementation is running there and thus what the
meaning of the bit is.  If the source is not under my control, I might
report the error to the owner of the source but there is nothing I can do
to debug it.  Thus, I don't care which implementation is used there and
what the bits mean.

Henk