Subject: [apps-discuss] CBOR: Decimal fractions

From: "Manger, James H" <[email-masked]>
Date: Fri, 7 Jun 2013 15:00:38 +1000

draft-bormann-cbor-01 can express a decimal fraction m * 10^e with a tagged array of e and m. This is a useful feature.

Describing this feature by referring to decimal64 in RFC6020 "YANG" only adds confusion. Why does the exponent have to be negative? Is the number m * 10^-e or m * 10^e? Can the exponent be 0 (which requires using the POSITIVE major type)? Can the mantissa be a bignum? How about the exponent?

Suggestion:

  2.3.3. Decimal Fractions

    A decimal fraction can precisely represent any number that
    can be expressed with decimal digits and a decimal point.
    It consists of an mantissa (m) and exponent (e), both of
    which are integers. The number represented is m * 10^e.
    A decimal fraction is represented in CBOR using tag 4
    followed by an array holding 2 items: the exponent,
    then the mantissa. The exponent and mantissa can use any
    CBOR integer representation, including Bignums.

    Example: 273.15 could be represented as 27315 * 10^-2:
      C4             -- tag, #4
         82          -- array, 2 items
            21       -- negative, -1-1 = -2
            19 6AB3  -- positive, 0x6AB3 = 27315
    
P.S. Swapping the order in the array to mantissa then exponent would be more intuitive, matching how they are written (eg 123e-4, 123x10^-4).

Should we complete the set with "Big Real"? Tag xx followed by an array of m and e, representing the number m * 2^e, where m (and e) can be a bignum.

--
James Manger

---

From: Carsten Bormann <[email-masked]>
Date: Sat, 8 Jun 2013 01:18:58 +0200

On Jun 7, 2013, at 07:00, "Manger, James H" <[email-masked]> wrote:

> Describing this feature by referring to decimal64 in RFC6020 "YANG" only adds confusion.

Well, it *was* modeled after RFC 6020, but we can move this to a comment later.

> Why does the exponent have to be negative?

The idea is that you already can send integers as integers; the decimal fractions format is just in support of fractions that cannot be represented exactly as floats (e.g., 1.1 = 11e-1 ≅ 0xfb3ff199999999999a).
(If you have a good use-case for non-negative exponents, I'd like to hear it -- maybe I just lack imagination.)

> Is the number m * 10^-e or m * 10^e?

10**e (as e is a negative number).
(We use C syntax plus "**" in the draft, so 10^e would be 10 XOR e.  Sorry about that...)

> Can the exponent be 0 (which requires using the POSITIVE major type)?

No.  In this case, just send the mantissa as an integer, no need to wrap it in a decimal fraction.

> Can the mantissa be a bignum? How about the exponent?

This needs to be clarified, as the current text just says "integer" and does not define whether a bignum is an integer.  (I would expect a bignum mantissa to be highly useful, but a use for a bignum exponent escapes me.)

>            19 6AB3  -- positive, 0x6AB3 = 27315

Much nicer example format.
We should adopt this...

> P.S. Swapping the order in the array to mantissa then exponent would be more intuitive, matching how they are written (eg 123e-4, 123x10^-4).

People who invent floating point representations typically keep the exponent where the MSBs would be, which in network byte order is sent first.
For processing the number, this order is also slightly more useful.
(For a state-limited streaming JSON-to-CBOR converter this is lots of fun anyway, as you still have to do the decimal-binary conversion.  For this, you already have to store the entire mantissa, so this does not cause any additional storage requirement.)

> Should we complete the set with "Big Real"? Tag xx followed by an array of m and e, representing the number m * 2^e, where m (and e) can be a bignum.

Not a bad idea.  So far, we tried to stick to IEEE 754 for floating point.  Their big binary floats are a bit complicated to use, though, so I haven't made up my mind whether they should be added or something like this (but probably not with bignum exponents).  One nice thing is that the big float doesn't have to cover the Infinities and NaN, because we can already represent those in shorter formats.

(Why didn't we stick with IEEE 754 for decimal: IEEE 754's decimal floats are complicated by the fact that they also keep the mantissa in decimal, using the famous 10-bit Cowlishaw declets for three decimal digits each.  What a guy.  A fun subject for an evening at the fireplace.)

Grüße, Carsten

---

From: "Manger, James H" <[email-masked]>
Date: Sat, 8 Jun 2013 21:58:11 +1000

> > draft-bormann-cbor-01 can express a decimal fraction m * 10^e with a
> tagged array of e and m. This is a useful feature.
> >

To me it looked like a partial ASN.1 REAL (in value notation):
  { mantissa xxx, base 2|10, exponent xxx }
 
> The idea is that you already can send integers as integers; the decimal
> fractions format is just in support of fractions that cannot be
> represented exactly as floats (e.g., 1.1 = 11e-1 ≅
> 0xfb3ff199999999999a).
> (If you have a good use-case for non-negative exponents, I'd like to
> hear it -- maybe I just lack imagination.)

A protocol has, say, a currency field; needs exact decimal fractions; specifies the field is always encoded as a CBOR decimal fraction; even though some currency amounts are integers. Could rewrite integers so exponent is negative (eg change 1e9 to 10000000000e-1), but why bother.

In Java, a CBOR decimal fraction maps well to a java.math.BigDecimal. A BigDecimal’s scale (equivalent to -exponent) can be negative or positive.

> This needs to be clarified, as the current text just says "integer" and
> does not define whether a bignum is an integer.  (I would expect a
> bignum mantissa to be highly useful, but a use for a bignum exponent
> escapes me.)

No need for a bignum exponent.

--
James Manger

> > P.S. Swapping the order in the array to mantissa then exponent would
> be more intuitive, matching how they are written (eg 123e-4, 123x10^-
> 4).
> 
> People who invent floating point representations typically keep the
> exponent where the MSBs would be, which in network byte order is sent
> first.
> (For a state-limited streaming JSON-to-CBOR converter this is lots of
> fun anyway, as you still have to do the decimal-binary conversion.  For
> this, you already have to store the entire mantissa, so this does not
> cause any additional storage requirement.)
> 
> > Should we complete the set with "Big Real"? Tag xx followed by an
> array of m and e, representing the number m * 2^e, where m (and e) can
> be a bignum.
> 
> Not a bad idea.  So far, we tried to stick to IEEE 754 for floating
> point.  Their big binary floats are a bit complicated to use, though,
> so I haven't made up my mind whether they should be added or something
> like this (but probably not with bignum exponents).  One nice thing is
> that the big float doesn't have to cover the Infinities and NaN,
> because we can already represent those in shorter formats.
> 
> (Why didn't we stick with IEEE 754 for decimal: IEEE 754's decimal
> floats are complicated by the fact that they also keep the mantissa in
> decimal, using the famous 10-bit Cowlishaw declets for three decimal
> digits each.  What a guy.  A fun subject for an evening at the
> fireplace.)
> 
> Grüße, Carsten

---

From: Phillip Hallam-Baker <[email-masked]>
Date: Sat, 8 Jun 2013 08:05:45 -0400

There are three IEEE 754 floating point decimal representations already.

JavaScript only uses IEEE 754 binary64.

If this is a standards effort I think there should be a really good reason
not to use the established standard. Especially so when it is outside IETF
field of expertise.

On Sat, Jun 8, 2013 at 7:58 AM, Manger, James H <
[email-masked]> wrote:

> A protocol has, say, a currency field; needs exact decimal fractions;
> specifies the field is always encoded as a CBOR decimal fraction; even
> though some currency amounts are integers. Could rewrite integers so
> exponent is negative (eg change 1e9 to 10000000000e-1), but why bother.
>
> In Java, a CBOR decimal fraction maps well to a java.math.BigDecimal. A
> BigDecimal’s scale (equivalent to -exponent) can be negative or positive.
>
>
> _______________________________________________
> apps-discuss mailing list
> [email-masked]
> https://www.ietf.org/mailman/listinfo/apps-discuss
>

-- 
Website: http://hallambaker.com/

---

From: Carsten Bormann <[email-masked]>
Date: Sat, 8 Jun 2013 17:47:58 +0200

>  { mantissa xxx, base 2|10, exponent xxx }

I liked the idea behind YANG decimals to only add what is not already there.
Integers can already be used to represent the case where the exponent is non-negative.
(IEEE 754 binary float is a different situation because it is a well-established standard.)

> A protocol has, say, a currency field; needs exact decimal fractions; specifies the field is always encoded as a CBOR decimal fraction;

Don't do that, then.
Maybe the text should be more explicit that decimalfraction is a complement to the existing integer capability.

> In Java, a CBOR decimal fraction maps well to a java.math.BigDecimal. A BigDecimal’s scale (equivalent to -exponent) can be negative or positive.

Mapping to existing libraries is a better argument.
However, instead of binary mantissa/base-10 exponent representation as in decimalfraction, some of the bigdecimal libraries might map even better to a representation that keeps the digits in the mantissa in decimal, too.
It is not hard to add such a type as a new tag to CBOR.
Likely, this type would use IEEE 754 decimal (which used to be called "densely-packed decimal" for a reason).

Decimalfraction is for the case where your arithmetic is in binary, but you do have a fractional scale that can be expressed using a negative base-10 exponent.  The example in the draft is about a temperature reading. Of course there could be separate meta-information somewhere else that says the reading is in centi-Kelvin, but it is often more convenient to keep the scale with the data.

An argument can be made that the format should not make restrictions that are somewhat arbitrary, and limiting the exponent for decimalfraction to a negative number may be slightly arbitrary for one specific application, while other applications benefit from it.  For java.math.BigDecimal, the current decimalfraction means that the sender needs to check if the scale is negative (= exponent is positive), and then multiply the mantissa with 10**-scale before encoding; the encoding would then be as an integer/bignum as with a zero scale (= zero exponent).  (There would be no need for a similar operation for decoding.)

My current view is that a full-featured decimal-with-all-frills would look more like IEEE 754 decimal, and the decimalfraction type, as inspired by RFC 6020, should stay minimal.  It is exactly the fraction type you would want to use with SenML, and it is a non-starter to put IEEE 754 decimals into a constrained sensor -- the declet conversion alone is almost as big as the entirety of CBOR.  But I'm also open to lifting the "exponent must be negative" restriction.

> No need for a bignum exponent.

Thanks for confirming this.

Grüße, Carsten