Subject: [core] Valid characters in SenML JSON (UTF-8 vs. ASCII)

From: Ari Keränen <[email-masked]>
Date: Tue, 20 Oct 2015 16:38:11 +0000

Hi CoRE WG folks,

Recently there has been couple of off-line discussions on whether the current draft text restricts the character set to be used with JSON SenML too much.

In particular, section 6 currently says [1]:

>    All of the data is UTF-8, but since this is for machine to machine
>    communications on constrained systems, only characters with code
>    points between U+0001 and U+007F are allowed which corresponds to the
>    ASCII[RFC0020] subset of UTF-8.

Some justification for this is given in section 11.2.1:

>    Specifically, only the ASCII
>    [RFC0020] subset of the UTF-8 characters are allowed.  This
>    simplifies implementation of very simple system and does not impose
>    any significant limitations as all this data is meant for machine to
>    machine communications and is not meant to be human readable.

However, it has been argued that e.g. place names with non 7-bit ASCII characters should be supported, which seems sensible.

Standard JSON can be UTF-8 -16 or even -32 [2]. However, for better interoperability (I-JSON), UTF-8 with without Surrogates or Noncharacters is recommended [3].

Moreover, there is difference what kind of values we allow for the generic String data ("sv") and what is allowed as value for the name parameters ("n" and "bn"). For names, there's currently additional restriction in the draft [4]:

>    The resulting concatenated name MUST consist only of characters out
>    of the set "A" to "Z", "a" to "z", "0" to "9", "-", ":", ".", or "_"
>    and it MUST start with a character out of the set "A" to "Z", "a" to
>    "z", or "0" to "9".  This restricted character set was chosen so that
>    these names can be directly used as in other types of URI including
>    segments of an HTTP path with no special encoding. 

We would now like to hear from WG participants if we should relax these requirements and allow, e.g., the I-JSON recommended "UTF-8 with without Surrogates or Noncharacters" in the string valued data. We probably want to exclude the quote character from the allowed values for easier parsing and not needing to escape that though.

Also, should we relax the name (and base name) restriction or do we want to keep simple interoperability (/concatenability) with URIs feature?

Finally, should we add a "binary blob" data type, using base64 encoded data, to accommodate for more exotic data. Perhaps instead of (relaxing) string values?

Cheers,
Ari

[1] https://tools.ietf.org/html/draft-jennings-core-senml-01#section-6
[2] https://tools.ietf.org/html/rfc7159#section-8.1
[3] https://tools.ietf.org/html/rfc7493#section-2.1
[4] https://tools.ietf.org/html/draft-jennings-core-senml-01#page-6

---

From: "Cullen Jennings (fluffy)" <[email-masked]>
Date: Tue, 27 Oct 2015 06:17:19 +0000

So the only places that I think we have an issue for hit stuff is the values that can go in the string values. For things like the names of the fields like v, bv, bt, t, etc, I think staying with current restrictions is just fine and makes things faster and easier.

On the topic of what data goes in the sv (string value), I think we need to discuss real world use cases. I have a hard time doing this because I have not seen very many real uses cases for this that would not be better handled by enums. Every time I have seen use cases in other things for UTF-8, it turned out a lot more than just that was needed, like lang strings and support of multiple values in different languages. I doubt that is where we want to go SENML so I’d like to propose we remove sv until we get better use cases for it. 

I suspect the best use case for it are going to be some weird sensor that just wants to send a bunch of raw binary values. For this, a binary blog would work out better than UTF-8 but we would need to send it in such a way that it was legal and easily parable inside the JSON, or XML or whatever it was being sent in. CBOR could probably directly support the binary blob, JSON and XML could base64 encode it. 

> On Oct 21, 2015, at 1:38 AM, Ari Keränen <[email-masked]> wrote:
> 
> Hi CoRE WG folks,
>

---

From: Andrew Mcgregor <[email-masked]>
Date: Mon, 26 Oct 2015 23:26:04 -0700

You really don't want to send binary blobs in a string that is supposedly
UTF8, because someone is going to try to validate it.  Better to encode it.

How about this for a case that really does need UTF8: suppose the sensor is
an RFID scanner, and the scanned object is someone's name badge.  It would
be nice if their name could be in the correct language.  Same goes for
other fields on a badge, and sensor metadata like location notes etc.

We could still base64 the UTF8 string, but that's just painful.

On 26 October 2015 at 23:17, Cullen Jennings (fluffy) <[email-masked]>
wrote:

>
> So the only places that I think we have an issue for hit stuff is the
> values that can go in the string values. For things like the names of the
> fields like v, bv, bt, t, etc, I think staying with current restrictions is
> just fine and makes things faster and easier.
>
> On the topic of what data goes in the sv (string value), I think we need
> to discuss real world use cases. I have a hard time doing this because I
> have not seen very many real uses cases for this that would not be better
> handled by enums. Every time I have seen use cases in other things for
> UTF-8, it turned out a lot more than just that was needed, like lang
> strings and support of multiple values in different languages. I doubt that
> is where we want to go SENML so I’d like to propose we remove sv until we
> get better use cases for it.
>
> I suspect the best use case for it are going to be some weird sensor that
> just wants to send a bunch of raw binary values. For this, a binary blog
> would work out better than UTF-8 but we would need to send it in such a way
> that it was legal and easily parable inside the JSON, or XML or whatever it
> was being sent in. CBOR could probably directly support the binary blob,
> JSON and XML could base64 encode it.
>
>
>
> > Recently there has been couple of off-line discussions on whether the
> current draft text restricts the character set to be used with JSON SenML
> too much.
> >
> > However, it has been argued that e.g. place names with non 7-bit ASCII
> characters should be supported, which seems sensible.
> >
> > Standard JSON can be UTF-8 -16 or even -32 [2]. However, for better
> interoperability (I-JSON), UTF-8 with without Surrogates or Noncharacters
> is recommended [3].
> >
> > Moreover, there is difference what kind of values we allow for the
> generic String data ("sv") and what is allowed as value for the name
> parameters ("n" and "bn"). For names, there's currently additional
> restriction in the draft [4]:
> >
> > We would now like to hear from WG participants if we should relax these
> requirements and allow, e.g., the I-JSON recommended "UTF-8 with without
> Surrogates or Noncharacters" in the string valued data. We probably want to
> exclude the quote character from the allowed values for easier parsing and
> not needing to escape that though.
> >
> > Also, should we relax the name (and base name) restriction or do we want
> to keep simple interoperability (/concatenability) with URIs feature?
> >
> > Finally, should we add a "binary blob" data type, using base64 encoded
> data, to accommodate for more exotic data. Perhaps instead of (relaxing)
> string values?
> >
> >
> > Cheers,
> > Ari
> >
> _______________________________________________
> core mailing list
> [email-masked]
> https://www.ietf.org/mailman/listinfo/core
>

-- 
Andrew McGregor | SRE | [email-masked] | +61 4 1071 2221

---

From: "Cullen Jennings (fluffy)" <[email-masked]>
Date: Tue, 27 Oct 2015 06:49:18 +0000

that’s a good use case. I just happened to have the specs for a particular RFID badge reader specs sitting on my computer and they are clearly capable of sending any binary blob - not just UTF8. I’m sure other might do other things but I think it hits the point that utf-8 string may not be enough for theses types of use case.

On Oct 27, 2015, at 3:26 PM, Andrew Mcgregor <[email-masked]<mailto:[email-masked]>> wrote:

You really don't want to send binary blobs in a string that is supposedly UTF8, because someone is going to try to validate it.  Better to encode it.

How about this for a case that really does need UTF8: suppose the sensor is an RFID scanner, and the scanned object is someone's name badge.  It would be nice if their name could be in the correct language.  Same goes for other fields on a badge, and sensor metadata like location notes etc.

On 26 October 2015 at 23:17, Cullen Jennings (fluffy) <[email-masked]<mailto:[email-masked]>> wrote:

> On Oct 21, 2015, at 1:38 AM, Ari Keränen <[email-masked]<mailto:[email-masked]>> wrote:
>
> Hi CoRE WG folks,
>

_______________________________________________
core mailing list
[email-masked]<mailto:[email-masked]>
https://www.ietf.org/mailman/listinfo/core

--
Andrew McGregor | SRE | [email-masked]<mailto:[email-masked]> | +61 4 1071 2221