Subject: [netmod] decimal64 canonical form

From: Andy Bierman <[email-masked]>
Date: Sun, 03 May 2009 19:54:31 -0700

Hi,

yang-05, sec. 9.3.2 says that leading and trailing zeros
are prohibited.

Perhaps the word 'extra' trailing zeros is intended here.
By this rule, even 0.0 is an invalid canonical form.

I do not think trailing zeros should be prohibited.
The exact fraction-digits value SHOULD be used
by the agent to make the precision of the number
clear every time it is used.

Think about the usage survey Phil posted awhile back.
Data types like percentage, currency, and probability
should all be printed with the complete number of
fraction digits.

That affects the canonical form of zero (0.0).
I suggest '0' instead of '0.0', '0.0000', etc.

Andy

---

From: Martin Bjorklund <[email-masked]>
Date: Mon, 04 May 2009 09:59:21 +0200 (CEST)

> By this rule, even 0.0 is an invalid canonical form.

You didn't quote the entire sentence.  It says:

   Leading and trailing zeros are prohibited, subject to the rule that
   there MUST be at least one digit before and after the decimal
   point.

This is they way it works in XSD, and I believe we have WG consensus
to do the same.

/martin

---

From: Andy Bierman <[email-masked]>
Date: Mon, 04 May 2009 06:39:07 -0700

> This is they way it works in XSD, and I believe we have WG consensus

I do not agree.
By this rule, percentage and currency will not
be printed correctly.  It is far more likely
that fixed-point numbers are printed this way
than with truncated trailing zeros.

This is a CLR -- saying agents MUST NOT return 3.00
for three dollars, etc. (Agents MUST return canonical form.)

It is hard to follow the logic of this WG sometimes.
Usually, XML or XSD is irrelevant and incomplete and that's
why YANG is different instead.  Other times, for no
apparent reason, YANG *must* follow exactly what XML
or XSD did, or the sky will fall in.

That's why I prefer to cite use-cases to support technical
changes.  At least that can be evaluated objectively,
unlike this "be like XSD when it suites us" argument.

Andy

---

From: Phil Shafer <[email-masked]>
Date: Mon, 4 May 2009 09:57:02 -0400

>It is hard to follow the logic of this WG sometimes.

IMHO the logic is something like this:
- if there's something that exists that does what we need
  then use it
- else if we can closely follow something existing
  then do
- else if we have to go our own way
  then do it and pay the price

There's a cost for inventing new technology, so we want to ensure
there's a matching benefit.  Here we are making something new but
trying to allow it to be expressed in something existing, thereby
reducing the impact and increasing the utility of the XSD output
that can be generated from YANG.

As far as use cases, your percentages would be expressed as "percent
99.94" or "percent 70.0", so I don't see an issue here (though I
voted to nuke the trailing ".0" since it too closely resembles
significant digits to me, but the WG saw the utility in allowing
something that can be expressed in XSD, so...).

I agree that money would not be adequately addressed.  Do we need
to invent a "retained-digits" to specify the number of digits to
the right of the decimal point that are retained in the canonical
form?  Given the rarity of netconf devices handling money, I'm not
sure this is needed.

Thanks,
 Phil

---

From: Andy Bierman <[email-masked]>
Date: Mon, 04 May 2009 07:41:34 -0700

> something that can be expressed in XSD, so...).

I am used to viewing spreadsheets.
I have never seen a spreadsheet truncate fixed-point numbers.
They are always printed with the specified number of decimal
places, like 99.94 or 70.00.

> form?  Given the rarity of netconf devices handling money, I'm not

What problem is being solved by forcing agents to remove 'extra'
zeros at the end of a string representing a decimal64 number?

It is wasted bandwidth?
We can't bear to send those extra 2 or 3 bytes on the wire?
That hardly fits with the rest of XML verbosity.

Is it to save operators from viewing redundant information?
Trailing zeros have meaning.
3.0 and 3.000 are not the same numbers.
Chopping off these zeros turns +/- 5 ten-thousandths
into +/- 5 hundredths.  For currency, it is more than unusual
to remove trailing zeros.

Is it to save work?
Removing these zeros is a little extra work for the agent,
and a lot of extra work for applications and operators
who need to refer back to the YANG file just to view
a decimal64 data type correctly.

Since selecting a large number of fraction digits dramatically
reduces the range of numbers (-9 to 9 in the worst case),
YANG data modelers will be very careful when using decimal64.
The requested number of digits is not going to be arbitrary.
The data modeler probably wants to see those digits.

Andy

---

From: Andy Bierman <[email-masked]>
Date: Mon, 04 May 2009 09:02:09 -0700

> something that can be expressed in XSD, so...).

Which part of XSD are we not reinventing here?
I cannot find any fixed point data types in XSD.
In scientific, business, and other areas where fixed point
data types are actually supported, they are never
rendered with truncated zeros.

I propose that the canonical form MUST include exactly the
number of fraction digits specified in the YANG file,
except the number zero, which is always represented as '0'
for all numeric data types in YANG.

Andy

---

From: Phil Shafer <[email-masked]>
Date: Mon, 4 May 2009 11:29:44 -0400

>zeros at the end of a string representing a decimal64 number?

Are you asking why we have canonical representations?  We do this
to maximize the ability of tools to process values without knowing
their type, for example, in an xpath expression or an xslt script.

>to remove trailing zeros.

Are you talking about significant digits?  For most of the world,
3 is identical to 3.000000.  And $1.99 is identical to $1.99000000000

Thanks,
 Phil

---

From: Andy Bierman <[email-masked]>
Date: Mon, 04 May 2009 10:02:00 -0700

> their type, for example, in an xpath expression or an xslt script.

No, I am asking we we are truncating zeros when
there are many use cases in which the zeros are important.

In every case there is no problem if the exact number
of significant digits are generated.  The manager can easily
remove any extra zeros if the user requests that these
numbers should be displayed that way.

> 3 is identical to 3.000000.  And $1.99 is identical to $1.99000000000

I disagree.
Did you get to $1.99 via rounding or some other mechanism?

I think you are mixing up the raw data at the protocol PDU layer
with the user-visible presentation layer in the NMS.

Andy

---

From: Wes Hardaker <[email-masked]>
Date: Tue, 05 May 2009 04:45:05 -0700

>>>>> On Mon, 04 May 2009 10:02:00 -0700, Andy Bierman <[email-masked]> said:

AB> Did you get to $1.99 via rounding or some other mechanism?

AB> I think you are mixing up the raw data at the protocol PDU layer
AB> with the user-visible presentation layer in the NMS.

You're both talking about two different use cases and trying to use one
datatype to meet both needs.

1) a number without any extra encoded information.  "1.0" really is the
   same as "1.000" if there is no precision information associated with
   it.  Go ask any language that supports floats if 1.0 == 1.000 and
   it'll return true.

2) a number with some level of implied precision associated with the
   data, where 1.000 really implies a different level of confidence
   associated with the calculation than 1.0.  For science this is
   frequently an important attribute to know and understand.  Though I
   would argue that if you take "1.000" out of context you really have
   lost all knowledge of why those trailing zeros were there and can't
   bet on the fact that it is promising precision out to that point.

   Shouldn't a number with a defined precision level get its own
   datatype that would encode the precision definitely along with it so
   that when you transfer it around that definition isn't lost?

And canonicalization of both of those cases are likely to have different
results, which is where you're running into trouble.

>From a data storage perspective and machine generic coding perspective,
the advantage of canonicalization is that I can expect and compare the
same data with a generic compare routine (memcmp) instead of needing to
know the type so that 1.0 really is different than 1.000 because I'm
functionally treating everything as a binary blob or string.  I think
that's already been said though.

-- 
"In the bathtub of history the truth is harder to hold than the soap,
 and much more difficult to find."  -- Terry Pratchett

---

From: Andy Bierman <[email-masked]>
Date: Tue, 05 May 2009 05:46:53 -0700

> that's already been said though.

The current proposal is that zero is '0' for all numeric types
except decimal64, which is encoded as '0.0' instead.  So your
memcmp test fails already.  '23' and '23.0' are not going to compare
correctly either.  If the manager saves '23.0' and the agent
then rejects any filter match on '23.0' (because it only will
match '23' instead), then have we solved the problem or just moved
it somewhere else?

Andy

---

From: Per Hedeland <[email-masked]>
Date: Tue, 05 May 2009 15:26:03 +0200

On 05/05/09 14:46, Andy Bierman wrote:
> except decimal64, which is encoded as '0.0' instead.

Hm, an alternate but as far as I know equally valid description is "the
current proposal is that the canonical form of zero is '0' for all
integer types and '0.0' for all decimal types". Also, I fail to see why
this difference is any more "interesting" than the same for '1' vs '1.0'
etc.

> So your

Is it a goal that it should be possible to do "implicit" value space
comparison via textual comparison between elements of different types?

FWIW, I don't see any particular value in the "strip trailing zeroes"
rule - i.e. "always print <fraction-digits> decimals" seems like an
equally reasonable definition of the canonical form.

--Per

---

From: Andy Bierman <[email-masked]>
Date: Tue, 05 May 2009 08:20:13 -0700

> comparison via textual comparison between elements of different types?

I think the point of having a canonical form is to allow
managers to compare values in the lexical space -- the
decoded XML, without any concept of the YANG semantics
associated with the XML (e.g., strcmp(a, b)).

The flaw with this plan is that it only works on a subset
of YANG data types, and the manager doesn't understand
the value-space (YANG definition), so it doesn't know which
XML nodes sent by the agent can be compared this way.

> equally reasonable definition of the canonical form.

thanks for phrasing it this way.
The choice we are making is which arbitrary algorithm
should be used to help support this flawed plan.
IMO, the majority of use cases for fixed-point numbers,
the user prefers to view all the fraction digits.

It is also possible for any manager (even dumb ones, which
are unaware of the YANG semantics) to remove extra zeros.
It is impossible for a dumb manager to add back any
zeros that the agent has already removed.

Andy

---

From: Per Hedeland <[email-masked]>
Date: Tue, 05 May 2009 18:23:18 +0200

> associated with the XML (e.g., strcmp(a, b)).

Yes of course - but values of *different types*? I can't even see how
any non-data-model-aware software could expect to be able to compare
anything other than different instances of the same XML element - nor
why it would be useful for it to be able to do so.

> are unaware of the YANG semantics) to remove extra zeros.

That would be completely broken IMO - if the manager isn't aware of the
data model, it could be removing critical information from some
application-specific string type. It can't just assume that it's a
numerical/decimal type from the fact that an instance of the textual
representation consists of only digits and a dot.

--Per Hedeland

---

From: Juergen Schoenwaelder <[email-masked]>
Date: Tue, 5 May 2009 18:48:35 +0200

On Tue, May 05, 2009 at 06:23:18PM +0200, Per Hedeland wrote:

[...]

1) Canonicalization is a type specific property.

2) We need canonicalization primarily so that tools comparing lets say
   different versions of a specific configuration can operate on the
   lexical respresentations.

/js

-- 
Juergen Schoenwaelder           Jacobs University Bremen gGmbH
Phone: +49 421 200 3587         Campus Ring 1, 28759 Bremen, Germany
Fax:   +49 421 200 3103         <http://www.jacobs-university.de/>

---

From: Andy Bierman <[email-masked]>
Date: Tue, 05 May 2009 09:56:58 -0700

> why it would be useful for it to be able to do so.

OK, for the same element the manager can compare
different instances of the same object without
knowing any semantics.

> representation consists of only digits and a dot.

You are right.
The manager could guess that '0' equals '0.0'
or <digits>.<digits> is a number, but it is only a guess.

Andy

---

From: Wes Hardaker <[email-masked]>
Date: Wed, 06 May 2009 10:31:37 -0700

>>>>> On Tue, 05 May 2009 05:46:53 -0700, Andy Bierman <[email-masked]> said:

AB> The current proposal is that zero is '0' for all numeric types
AB> except decimal64, which is encoded as '0.0' instead.
AB> So your memcmp test fails already.

No, the memcmp would still succeed because agents would be returning
canonical forms of each variable in the tree in identical ways.  A
manager would never generically compare something in one part of the
tree of a one type (int) to another part of the tree in another type
(decimal64).

That being said, managers should ideally understand the data types that
they're trying to look at.  But generic "fetch and store" mechanisms
can't always do this.  I've worked with people that can't get broken
MIBs to load, so they use generic data.  SNMP has an advantage here
since the data is sent over in typed format and the storage/printing
engines don't need the yang data type around to understand the
fundamental nature of the value.  This isn't true with netconf (I'm not
arguing it should be; I'm just pointing it out).

Operators *will* try and hack around cases where they don't have the
yang module available or the yang module is not parsable by their tools.
-- 
"In the bathtub of history the truth is harder to hold than the soap,
 and much more difficult to find."  -- Terry Pratchett

---

From: Andy Bierman <[email-masked]>
Date: Wed, 06 May 2009 11:55:58 -0700

> tree of a one type (int) to another part of the tree in another type

This can happen as a side effect of subtree and/or XPath filtering,
depending on the filter and data model contents.

> yang module available or the yang module is not parsable by their tools.

IMO, this is an NMS presentation issue.
The NMS can let the user format unknown leafs
however they want.  The user can say "treat this
as an int32, or a decimal64 w/ 4 fraction digits", or whatever.

If the application has the YANG module, then it can
figure out how to compare values.  Just assuming
the agent correctly returned the canonical form
may not always work.

Andy