Subject: [netmod] Justification for decimal64 over string for floating point values in geo location data?

From: Christian Hopps <[email-masked]>
Date: Tue, 7 Jul 2020 07:06:20 -0400

I received feedback in my YANG doctor review (thanks Mahesh) regarding the use of decimal64 for most of the values in the geo location grouping (https://tools.ietf.org/html/draft-ietf-netmod-geo-location-04). In my comparison sections I note that some precision (at the very extremes) may be lost when converting from other geo location formats that use string (or double for w3c) to decimal64. Given that mention of loss of extreme precision, the reviewer was asking if some justification for the decimal64 should be given in the document.

What are the advantages to using decimal64 for floating point numbers vs using a string with a pattern "[0-9]+(\.[0-9]+)?" (convert that to yang pattern language). The advantage of using a string is that the precision of the value is not restricted by the model. Does the YANG decimal64 values have a concise binary format that can be more efficiently transported or stored in binary form? If so is this the only advantage, and is it enough of one to limit the precision in the model?

It's definitely worth noting that the precision of the decimal64 values seem vastly adequate for geo location data (e.g., for Cartesian coordinates and height values which are measured in meters the fractional digits is 6 which means the surface could be up to 9 billion kilometers large (or away from for height) and the precision is to the micrometer. For ellipsoidal coordinates there are 12 fractional digits for the degrees.

Thanks,
Chris.

---

From: Juergen Schoenwaelder <[email-masked]>
Date: Tue, 7 Jul 2020 13:24:50 +0200

Precision often means different things to different people. Here is my
take:

- Floating point numbers have almost always rounding errors. And
  floating point numbers use binary fractions, a decimal fraction like
  1.0 has no precise representation as a binary fraction. Type 0.1 +
  0.2 into python or haskell or any other language that gives you bare
  floating point numbers and enjoy the result.

- Fixed precision decimal numbers do not have rounding errors since
  they are essentially scaled integers and hence they are precise as
  long as calculations stay within the range.

- Floating point numbers can cover a large number space (from very
  tiny to really big), fixed precision decimal numbers are much more
  restrictive.

- In XML and JSON, numbers are rendered in strings that likely do not
  look much different if its a decimal64 or a float or ... If you really
  care about size, use a binary encoding like CBOR.

/js

> _______________________________________________
> netmod mailing list
> [email-masked]
> https://www.ietf.org/mailman/listinfo/netmod

-- 
Juergen Schoenwaelder           Jacobs University Bremen gGmbH
Phone: +49 421 200 3587         Campus Ring 1 | 28759 Bremen | Germany
Fax:   +49 421 200 3103         <https://www.jacobs-university.de/>

---

From: Ladislav Lhotka <[email-masked]>
Date: Tue, 7 Jul 2020 13:51:25 +0200

>   care about size, use a binary encoding like CBOR.

I know nothing about geo-location formats, but I suspect that the string
representation is based on some assumptions regarding the underlying
numeric type. So one option might be to define a new type derived from
"string", and specify these assumptions in the description.

Lada

-- 
Ladislav Lhotka
Head, CZ.NIC Labs
PGP Key ID: 0xB8F92B08A9F76C67

---

From: Christian Hopps <[email-masked]>
Date: Tue, 7 Jul 2020 07:53:19 -0400

> On Jul 7, 2020, at 7:24 AM, Juergen Schoenwaelder <[email-masked]> wrote:
> 

So do you think it's enough to just use decimal64, and not justify it's use over strings? I don't necessarily think we need to talk to it in the document, but it was raised in the review so I figured some discussion on the list was at least merited.

While I haven't done any binary stuff with YANG, I think it's definitely usable that way (e.g., to define binary APIs in software where you probably wouldn't want to be using XML or JSON).

Thanks,
Chris.

---

From: Juergen Schoenwaelder <[email-masked]>
Date: Tue, 7 Jul 2020 14:18:21 +0200

On Tue, Jul 07, 2020 at 07:53:19AM -0400, Christian Hopps wrote:
> 
> So do you think it's enough to just use decimal64, and not justify it's use over strings?

I do not know what 'strings' are here.

In the xml and json encodings, everything finally has a string
representation. But what matters is how the strings are interpreted
since computers internally often do not use strings when it comes to
numbers or addresses or ...

/js

---

From: Christian Hopps <[email-masked]>
Date: Tue, 7 Jul 2020 08:27:19 -0400

> On Jul 7, 2020, at 8:18 AM, Juergen Schoenwaelder <[email-masked]> wrote:
> 

Mentioned in the earlier mail

instead of

    "decimal64"

use

    "type string { pattern '[0-9]+(\.[0-9]+)?'; }"

> numbers or addresses or ...

But they can operate with arbitrarily precise numbers (e.g., python mpmath, or even decimal with some arbitrarily large precision specified).

Thanks,
Chris.

---

From: Juergen Schoenwaelder <[email-masked]>
Date: Tue, 7 Jul 2020 14:30:24 +0200

>     "type string { pattern '[0-9]+(\.[0-9]+)?'; }"

And then everybody implements what he/she likes? That would be a big
step backward since every implementation will then interpret the
numbers differently.

/js

---

From: Christian Hopps <[email-masked]>
Date: Tue, 7 Jul 2020 09:24:31 -0400

> On Jul 7, 2020, at 8:30 AM, Juergen Schoenwaelder <[email-masked]> wrote:
> 

I dislike having to specify arbitrary limits b/c of the type. I think it would be useful to have integer and real number support that allowed for specifying "at least" precision, but not forcing the model to specify an "at most".

In generally I dislike many of the specified semantic restrictions I find in YANG models, they seem to be the most oft-cited examples of when a "backward incompatible" change might need to be made to a model.

BTW, I think decimal64 is fine for this use-case, I mainly asked on the list b/c I'm curious about the more general use-case.

Thanks,
Chris.

---

From: Juergen Schoenwaelder <[email-masked]>
Date: Tue, 7 Jul 2020 15:57:34 +0200

> I dislike having to specify arbitrary limits b/c of the type. I think it would be useful to have integer and real number support that allowed for specifying "at least" precision, but not forcing th…

You can have defined limits or undefined limits with implementation
specific limits (only rarely you find implementations that de-facto
have no limits).

> In generally I dislike many of the specified semantic restrictions I
> find in YANG models, they seem to be the most oft-cited examples of
> when a "backward incompatible" change might need to be made to a
> model.
 
ASN.1 had INTEGER, a type of unlimited precision. In the early SNMP
days things started to fail to interoperate when you hit the 16-bit
limit. JSON (much newer) has numbers that start to become interesting
when you need more precision, a simple 64-bit counter starts falls
apart in JSON.

I can't tell how many of the "backward incompatible" changes are due
to picking too restrictive types.

/js

---

From: Ladislav Lhotka <[email-masked]>
Date: Tue, 7 Jul 2020 16:24:03 +0200

> when you need more precision, a simple 64-bit counter starts falls

If you mean "counter64" in the ietf-yang-types module, then it doesn't
fall apart in JSON, provided that the rules of RFC 7951 are followed.

Lada

---

From: Juergen Schoenwaelder <[email-masked]>
Date: Tue, 7 Jul 2020 16:33:24 +0200

> fall apart in JSON, provided that the rules of RFC 7951 are followed.

I was alluding to plain JSON where you encode a number as a number
until you realize that some numbers must be encoded as strings to
avoid surprises. RFC 7951 does this for YANG defined data.

/js

---

From: David Spakes <[email-masked]>
Date: Tue, 7 Jul 2020 10:40:39 -0400 (EDT)

Like Lada, I know nothing about geo-location formats.  But this
discussion sounds familiar because it keeps coming up time and
again.  There are arguments for and against decimal64, and people
have before expressed a need for a variable-precision type.

I proposed the following compromise/solution in 2009 and again in 2014.
Now, I'll propose it again: a derived type that solidifies the
position/preservation of decimal64 and builds on it to deliver the
desired feature.

    typedef real {
      type union {
        type decimal64 {
          fraction-digits 18;
          range "-0.999999999999999999 .. 0.999999999999999999";
        }
        type decimal64 {
          fraction-digits 17;
          range "-9.99999999999999999 .. 9.99999999999999999";
        }
        type decimal64 {
          fraction-digits 16;
          range "-99.9999999999999999 .. 99.9999999999999999";
        }
        type decimal64 {
          fraction-digits 15;
          range "-999.999999999999999 .. 999.999999999999999";
        }
        type decimal64 {
          fraction-digits 14;
          range "-9999.99999999999999 .. 9999.99999999999999";
        }
        type decimal64 {
          fraction-digits 13;
          range "-99999.9999999999999 .. 99999.9999999999999";
        }
        type decimal64 {
          fraction-digits 12;
          range "-999999.999999999999 .. 999999.999999999999";
        }
        type decimal64 {
          fraction-digits 11;
          range "-9999999.99999999999 .. 9999999.99999999999";
        }
        type decimal64 {
          fraction-digits 10;
          range "-99999999.9999999999 .. 99999999.9999999999";
        }
        type decimal64 {
          fraction-digits 9;
          range "-999999999.999999999 .. 999999999.999999999";
        }
        type decimal64 {
          fraction-digits 8;
          range "-9999999999.99999999 .. 9999999999.99999999";
        }
        type decimal64 {
          fraction-digits 7;
          range "-99999999999.9999999 .. 99999999999.9999999";
        }
        type decimal64 {
          fraction-digits 6;
          range "-999999999999.999999 .. 999999999999.999999";
        }
        type decimal64 {
          fraction-digits 5;
          range "-9999999999999.99999 .. 9999999999999.99999";
        }
        type decimal64 {
          fraction-digits 4;
          range "-99999999999999.9999 .. 99999999999999.9999";
        }
        type decimal64 {
          fraction-digits 3;
          range "-999999999999999.999 .. 999999999999999.999";
        }
        type decimal64 {
          fraction-digits 2;
          range "-9999999999999999.99 .. 9999999999999999.99";
        }
        type decimal64 {
          fraction-digits 1;
          range "-99999999999999999.9 .. 99999999999999999.9";
        }
        type enumeration {
          enum overflow;
          enum underflow;
        }
      }
      description
        "The real type defines a large, finite set of real numbers with
         varying degrees of magnitude and precision.  An object of type
         real is able to express configuration and state data as any of
         the real numbers in the set.  The real type is suitable for
         applications that deal with ratios in which the decimal point is
         not fixed to a single position.

         Positive numbers larger than 99999999999999999.9 in state data
         SHALL be represented as an overflow.  The enumerated value
         overflow is not valid for an <edit-config> operation.

         Negative numbers larger than -99999999999999999.9 in state data
         SHALL be represented as an underflow.  The enumerated value
         underflow is not valid for an <edit-config> operation.

         Real numbers between -99999999999999999.9 and 99999999999999999.9
         (inclusive) having a combination of magnitude and precision that
         falls outside of one of the range substatements in the union--
         including irrational numbers such as 'pi'--may only be
         represented by the real type through rounding or truncation.
         For example, pi could be rounded (up) to 3.14159265358979324 or
         truncated to 3.14159265358979323 to preserve as much of the
         precision information as possible.  Note that an application
         could choose to round (down) the value of pi to 3.14 or truncate
         the value of pi to 3.14.  This specification does not define
         rules for the rounding or truncating of such values, considering
         these decisions to be application-specific.";
    }

David Spakes

> "string", and specify these assumptions in the description.

-------------------------------------------------------------
  David Spakes                       email:   [email-masked]
  SNMP Research                      voice:   +1 865 573 1434
  3001 Kimberlin Heights Road          fax:   +1 865 573 9197
  Knoxville, TN  37920-9716  USA      http://www.snmp.com
-------------------------------------------------------------

---

From: Ladislav Lhotka <[email-masked]>
Date: Tue, 7 Jul 2020 16:59:43 +0200

> position/preservation of decimal64 and builds on it to deliver the

This type might be much more useful if the effective union member could
be determined from the data. It is often important to differentiate
between 1.23 and 1.230, but this wouldn't be possible here, due to the
canonical form for "decimal64" (sec. 9.3.2 in RFC 7950).

Lada

---

From: "Rob Wilton (rwilton)" <[email-masked]>
Date: Wed, 8 Jul 2020 16:35:06 +0000

[As an individual]

I agree with Juergen that in many configuration cases, using decimal64 is better/safer than binary float/double.  However, there are other scenarios, such as operational data coming from sensors, where float/double is probably more appropriate/useful, hence I would still like to see the next version of YANG supporting float/double, possibly restricted to operational data only.

For configuration, with regards to the rounding errors alluded to below, I do have some sympathy with Chris's suggestion of support for arbitrary precision decimal numbers.  It seems that more and more languages have native support for arbitrary precision decimal maths.  I note that CBOR also has an encoding for them, and a JSON/XML encoding of them is seemingly trivial.

Regards,
Rob

> -----Original Message-----
> From: netmod <[email-masked]> On Behalf Of Juergen Schoenwaelder
> Sent: 07 July 2020 12:25
> To: Christian Hopps <[email-masked]>
> Cc: NetMod WG <[email-masked]>
> Subject: Re: [netmod] Justification for decimal64 over string for floating
> point values in geo location data?
> 
> > I received feedback in my YANG doctor review (thanks Mahesh) regarding
> the use of decimal64 for most of the values in the geo location grouping
> (https://tools.ietf.org/html/draft-ietf-netmod-geo-location-04). In my
> comparison sections I note that some precision (at the very extremes) may
> be lost when converting from other geo location formats that use string
> (or double for w3c) to decimal64. Given that mention of loss of extreme
> precision, the reviewer was asking if some justification for the decimal64
> should be given in the document.
> >
> > What are the advantages to using decimal64 for floating point numbers vs
> using a string with a pattern "[0-9]+(\.[0-9]+)?" (convert that to yang
> pattern language). The advantage of using a string is that the precision
> of the value is not restricted by the model. Does the YANG decimal64
> values have a concise binary format that can be more efficiently
> transported or stored in binary form? If so is this the only advantage,
> and is it enough of one to limit the precision in the model?
> >
> > It's definitely worth noting that the precision of the decimal64 values
> seem vastly adequate for geo location data (e.g., for Cartesian
> coordinates and height values which are measured in meters the fractional
> digits is 6 which means the surface could be up to 9 billion kilometers
> large (or away from for height) and the precision is to the micrometer.
> For ellipsoidal coordinates there are 12 fractional digits for the
> degrees.
> >
> > Thanks,
> > Chris.
> 
> 
>

---

From: Juergen Schoenwaelder <[email-masked]>
Date: Wed, 8 Jul 2020 18:44:12 +0200

unknown precision != arbitrary precision

/js

---

From: "Rob Wilton (rwilton)" <[email-masked]>
Date: Wed, 8 Jul 2020 16:53:52 +0000

> From: Juergen Schoenwaelder <[email-masked]>
> Sent: 08 July 2020 17:44
> To: Rob Wilton (rwilton) <[email-masked]>
> Cc: Christian Hopps <[email-masked]>; NetMod WG <[email-masked]>
[RW] 

I agree.  But don't follow the point that you are making.  Please can you clarify.

Regards,
Rob

> > I agree with Juergen that in many configuration cases, using decimal64
> is better/safer than binary float/double.  However, there are other
> scenarios, such as operational data coming from sensors, where
> float/double is probably more appropriate/useful, hence I would still like
> to see the next version of YANG supporting float/double, possibly
> restricted to operational data only.
> >
> > For configuration, with regards to the rounding errors alluded to below,
> I do have some sympathy with Chris's suggestion of support for arbitrary
> precision decimal numbers.  It seems that more and more languages have
> native support for arbitrary precision decimal maths.  I note that CBOR
> also has an encoding for them, and a JSON/XML encoding of them is
> seemingly trivial.
> >
> > Regards,
> > Rob
> >
> >

---

From: Juergen Schoenwaelder <[email-masked]>
Date: Wed, 8 Jul 2020 19:32:18 +0200

> I agree.  But don't follow the point that you are making.  Please can you clarify.

To me

   type string;
   pattern "[0-9]+(\.[0-9]+)?";

means unknown precision - everybody implements what seems convenient.

/js

---

From: Carsten Bormann <[email-masked]>
Date: Wed, 8 Jul 2020 19:45:54 +0200

> means unknown precision - everybody implements what seems convenient.

It also means that there is no way to convert this text string into a number when using YANG-CBOR.

Grüße, Carsten

---

From: "Rob Wilton (rwilton)" <[email-masked]>
Date: Thu, 9 Jul 2020 10:27:28 +0000

> Sent: 08 July 2020 18:32
[RW] 

Ah, Okay.  Yes, I agree.  I think that there are already too many types that get encoded as a string in YANG.

My suggestion was to consider for the next version of YANG to potentially add two new native YANG types:
 - Binary float and binary double native YANG types, possibly restricted to operational values only.
 - A new arbitrary-precision-decimal native YANG type.

Regards,
Rob

---

From: "Don Fedyk" <[email-masked]>
Date: Thu, 9 Jul 2020 08:19:11 -0400

+1
(Definitely too many numeric types encoded as strings ) Don

[RW] 

Ah, Okay.  Yes, I agree.  I think that there are already too many types that
get encoded as a string in YANG.

My suggestion was to consider for the next version of YANG to potentially
add two new native YANG types:
 - Binary float and binary double native YANG types, possibly restricted to
operational values only.
 - A new arbitrary-precision-decimal native YANG type.

Regards,
Rob