Subject: [apps-discuss] draft-pbryan-zyp-json-pointer: name syntax for non-ASCII

From: Julian Reschke <[email-masked]>
Date: Mon, 21 Nov 2011 15:12:54 +0100

Hi there.

In 
<https://tools.ietf.org/html/draft-pbryan-zyp-json-pointer-02#section-3> 
we have:

    A JSON Pointer is a sequence of zero or more reference tokens, each
    prefixed by a "/" (%x2F) character.  Each reference token is a
    sequence of unreserved and/or percent-encoded characters, per
    [RFC3986].

    json-pointer = *( "/" reference-token )
    reference-token = *( unreserved / pct-encoded )

    Characters in reference tokens that are not unreserved SHOULD be
    percent-encoded, per [RFC3986], and MUST be so encoded as "%2F" if
    the character is "/" to avoid being interpreted as a reference token
    prefix.

    It is an error condition if a JSON Pointer does not conform to this
    syntax.

This doesn't seem to consider the case where the reference token 
contains non-ASCII characters, which can happen with JSON.

There seem to be to obvious ways to address this:

(1) Allow non-ASCII characters in the pointer (which would make the 
pointers be more IRI-like, and I think that's consistent with XPath), or

(2) Require UTF-8 encoding.

I believe (1) makes more sense here.

Best regards, Julian

---

From: "Paul C. Bryan" <[email-masked]>
Date: Mon, 21 Nov 2011 11:24:23 -0800

RFC 3986 prescribes encoding characters in UTF-8 and percent-encoding
non-unreserved characters. Furthermore, JSON by definition uses Unicode
for all string values (most often encoding in UTF-8). Do these not
address the issue?

Paul

On Mon, 2011-11-21 at 15:12 +0100, Julian Reschke wrote:

> _______________________________________________
> apps-discuss mailing list
> [email-masked]
> https://www.ietf.org/mailman/listinfo/apps-discuss

---

From: Bjoern Hoehrmann <[email-masked]>
Date: Mon, 21 Nov 2011 20:37:05 +0100

>for all string values (most often encoding in UTF-8). Do these not

RFC 3986 only defines the character encoding for the host component, in
other components it just defines that %C3 refers to the octet 0xC3. So,
if you want that %C3%B6 refers to an "ö", then you have to define that.
-- 
Björn Höhrmann · mailto:[email-masked] · http://bjoern.hoehrmann.de
Am Badedeich 7 · Telefon: +49(0)160/4415681 · http://www.bjoernsworld.de
25899 Dagebüll · PGP Pub. KeyID: 0xA4357E78 · http://www.websitedev.de/

---

From: Julian Reschke <[email-masked]>
Date: Mon, 21 Nov 2011 20:43:58 +0100

On 2011-11-21 20:24, Paul C. Bryan wrote:
> ...

Not really -- RFC 3986 doesn't really require it; it just says it should 
be the default for new schemes.

So again, do you want the ability to use non-ASCII characters in JSON 
pointers without escaping? (I think that would be the right approach.)

Best regards, Julian

---

From: "Paul C. Bryan" <[email-masked]>
Date: Mon, 21 Nov 2011 11:54:57 -0800

I'm interpreting section 2.5 of RFC 3896:

> When a new URI scheme defines a component that represents textual data
> consisting of characters from the Universal Character Set [UCS], the
> data should first be encoded as octets according to the UTF-8
> character encoding [STD63]; then only those octets that do not
> correspond to characters in the unreserved set should be
> percent-encoded.  For example, the character A would be represented as
> "A", the character LATIN CAPITAL LETTER A WITH GRAVE would be
> represented as "%C3%80", and the character KATAKANA LETTER A would be
> represented as "%E3%82%A2".

Is this not applicable?

Paul

---

From: "Paul C. Bryan" <[email-masked]>
Date: Mon, 21 Nov 2011 11:59:59 -0800

> pointers without escaping? (I think that would be the right approach.)

Presuming section 2.5 is not applicable (or cannot be made applicable by
reference), then we would need to explicitly specify an encoding. Since
JSON is Unicode by definition, following the recommendation set forth in
section 2.5 to encode as UTF-8 and percent-encode each non-unreserved
octet would seem most appropriate.

Paul

---

From: Julian Reschke <[email-masked]>
Date: Mon, 21 Nov 2011 21:02:00 +0100

>> represented as "%E3%82%A2".

No.

For starters, we are not defining a URI scheme at all.

We're just re-using percent-encoding, and that is about mapping octets 
to characters. We still need to state what the octets are.

Best regards, Julian

---

From: Julian Reschke <[email-masked]>
Date: Mon, 21 Nov 2011 21:06:17 +0100

> octet would seem most appropriate.

Yes, that would work.

However, why encode those characters in the first place? Where does the 
requirement that the pointer needs to be all-ASCII come from?

Best regards, Julian

---

From: Bjoern Hoehrmann <[email-masked]>
Date: Mon, 21 Nov 2011 21:09:18 +0100

>> represented as "%E3%82%A2".

If I recommended "When writing a new crime novel, reveal the perpetrator
towards the end" then the author would still have to write the part, and
the reader would still have to read through the novel to find out where
the perpetrator is revealed, since the author might not be following the
recommendation. Besides, the draft does not define a new scheme. This is
just a matter of using the right incantation, but I don't know why, say,
"/Björn" isn't allowed, so I can't make a good suggestion at the moment.
-- 
Björn Höhrmann · mailto:[email-masked] · http://bjoern.hoehrmann.de
Am Badedeich 7 · Telefon: +49(0)160/4415681 · http://www.bjoernsworld.de
25899 Dagebüll · PGP Pub. KeyID: 0xA4357E78 · http://www.websitedev.de/

---

From: "Paul C. Bryan" <[email-masked]>
Date: Mon, 21 Nov 2011 12:09:49 -0800

The intent is to allow a JSON Pointer to be expressed as a JSON string
value as well as a URI fragment identifier. The latter is the most
significant driver for URI percent-encoding.

Paul

---

From: Frank Ellermann <[email-masked]>
Date: Mon, 21 Nov 2011 21:10:02 +0100

On 21 November 2011 21:02, Julian Reschke wrote:

> We're just re-using percent-encoding, and that is about mapping octets to
> characters. We still need to state what the octets are.

Maybe copy and paste the simple RFC 3987 idea, that gives you UTF-8 (outside
of DNS hostnames in IRIs).  Unless you want or need "raw" octets for other
purposes in this draft, but so far I think you don't.

-Frank

---

From: Julian Reschke <[email-masked]>
Date: Mon, 21 Nov 2011 21:12:44 +0100

> significant driver for URI percent-encoding.

Well, you could use it as fragment identifier (or otherwise URI 
component) by UTF-8-percent-escaping.

The question is whether that use case requires them to be all ASCII 
every else, such as in a JSON patch document.

Best regards, Julian

---

From: "Paul C. Bryan" <[email-masked]>
Date: Mon, 21 Nov 2011 12:14:20 -0800

Point taken. The intent of SHOULD in the text was to allow "/Björn" as a
JSON Pointer in a JSON string. Sounds like I should have separate
sections for expressing a JSON Pointer in JSON string vs. URI fragment,
and have clearer rules regarding the encoding of such values.

Paul

---

From: Mark Nottingham <[email-masked]>
Date: Tue, 22 Nov 2011 07:55:47 +1100

+1 to Julian here -- there's no reason why non-ASCII chars need to be percent-encoded when they occur inside a JSON document, only when they're in a URI (or similar context).

Cheers,

On 22/11/2011, at 7:12 AM, Julian Reschke wrote:

> Well, you could use it as fragment identifier (or otherwise URI component) by UTF-8-percent-escaping.
> 
> The question is whether that use case requires them to be all ASCII every else, such as in a JSON patch document.
> 
> Best regards, Julian

--
Mark Nottingham   http://www.mnot.net/

---

From: "Paul C. Bryan" <[email-masked]>
Date: Mon, 21 Nov 2011 13:51:09 -0800

Okay, so I'll write-up separate sections for JSON string values and URI
fragment identifiers. Any objections?

Paul

---

From: Mark Nottingham <[email-masked]>
Date: Tue, 22 Nov 2011 10:43:30 +1100

The usual approach to this sort of thing is to define the "canonical" way to do it; i.e., json pointers *are* unicode strings; then you can define encodings into various environments (like URIs).

In this case, it'd probably be good enough to say that json pointers are unicode strings, but that when they need to be in ASCII environments (like URIs) they get UTF-8'ed and then percent-escaped.

Cheers,

On 22/11/2011, at 8:51 AM, Paul C. Bryan wrote:

> Okay, so I'll write-up separate sections for JSON string values and URI fragment identifiers. Any objections?
> 
> Paul
>

---

From: "Paul C. Bryan" <[email-masked]>
Date: Mon, 21 Nov 2011 16:56:00 -0800

And of course, if a pointer references a member name with a slash, then
%25 MUST be used...

Paul

---

From: "Paul C. Bryan" <[email-masked]>
Date: Mon, 21 Nov 2011 16:57:23 -0800

On Mon, 2011-11-21 at 16:56 -0800, Paul C. Bryan wrote:

> And of course, if a pointer references a member name with a slash,
> then %25 MUST be used...

s/%25/%2F/

---

From: "Martin J. Dürst" <[email-masked]>
Date: Tue, 22 Nov 2011 18:09:04 +0900

On 2011/11/22 8:43, Mark Nottingham wrote:
> In this case, it'd probably be good enough to say that json pointers are unicode strings,

Up to here, this makes a ton of sense.

> but that when they need to be in ASCII environments (like URIs) they get UTF-8'ed and then percent-escaped.

This would mean that e.g. in a Java program that for some reason is kept 
in US-ASCII, I'd have to use %-encoding. This doesn't make sense to me 
at all.

So I'd say that json pointers are escaped according to the conventions 
of the substrate that carries them when needed (e.g. pure ASCII, or 
other kinds of encodings that can't handle the whole Unicode range).

Then for json pointers as fragment identifiers, I'd mention that where 
necessary (e.g. for URIs), the convention for converting from IRIs to 
URIs (see RFC 3987) applies.

By the way, I don't see a need to escape "/" at all in a fragment 
identifier. "/" is plain and simply allowed in fragment identifiers. 
Please see http://tools.ietf.org/html/rfc3986#section-3.5. Of course, 
it's not forbidden to escape "/", so software that is interpreting a 
fragment identifier has to make sure it does the right thing.

Regards,    Martin.

---

From: Julian Reschke <[email-masked]>
Date: Tue, 22 Nov 2011 10:13:12 +0100

On 2011-11-22 10:09, "Martin J. Dürst" wrote:
>> The usual approach to this sort of thing is to define the "canonical"
>> way to do it; i.e., json pointers *are* unicode strings; then you can
>> define encodings into various environments (like URIs).
>>

+1 up to here.

> fragment identifier has to make sure it does the right thing.

The escaping of "/" refers to the fact that "/" is used as delimiter in 
the pointer syntax (so if you have "/" in a token you need to escape it).

Speaking of fragment identifiers -- sounds like somebody is working on a 
fragment identifier syntax for the JSON media type???

Best regards, Julian

---

From: Carsten Bormann <[email-masked]>
Date: Tue, 22 Nov 2011 10:33:47 +0100

> +1 to Julian here -- there's no reason why non-ASCII chars need to be percent-encoded when they occur inside a JSON document, only when they're in a URI (or similar context).

OK, folks, let's see where that leads to.

JSON document:
{"Bjørn/Carsten": "Fritz"}

JSON pointer:
"/Bjørn%2FCarsten"

Multi-segment URI-encoded version of this JSON pointer:

/Bj%C3%B8rn%252FCarsten

(For reference: Plain URI-encoded version:

%2FBj%C3%B8rn%252FCarsten
)

If this is what you want, please document it thoroughly, clearly, painfully redundantly.
Dozens of examples, including examples that show how to do it wrong.

With this kind of dual-layer percent-encoding, we are deeply in quoting hell, and implementers will need all the help we can give them (and they will still get it wrong).

Grüße, Carsten

---

From: Graham Klyne <[email-masked]>
Date: Tue, 22 Nov 2011 11:26:13 +0000

I spotted this discussion and was reminded that one of the older URI specs had 
some explicit discussion of characters and octets and encoding.  I recall a line 
that looked something like this:

   URI characters -> URI octets -> URI octets %-encoded to US-ASCII

but I can no longer find it (quickly).  But http://www.ietf.org/rfc/rfc3986.txt 
sections 1.2 and section 2 (esp. intro) address some of the issues.  The point 
being that character encoding to octets is a separate concern from %-encoding to 
URI (or IRI) on-the-wire characters.

#g
--

On 22/11/2011 09:09, "Martin J. Dürst" wrote:
>> The usual approach to this sort of thing is to define the "canonical" way to
>> do it; i.e., json pointers *are* unicode strings; then you can define
>> encodings into various environments (like URIs).
>>
> This would mean that e.g. in a Java program that for some reason is kept in
> US-ASCII, I'd have to use %-encoding. This doesn't make sense to me at all.
>
> So I'd say that json pointers are escaped according to the conventions of the
> substrate that carries them when needed (e.g. pure ASCII, or other kinds of
> encodings that can't handle the whole Unicode range).
>
> Then for json pointers as fragment identifiers, I'd mention that where necessary
> (e.g. for URIs), the convention for converting from IRIs to URIs (see RFC 3987)
> applies.
>
> By the way, I don't see a need to escape "/" at all in a fragment identifier.
> "/" is plain and simply allowed in fragment identifiers. Please see
> http://tools.ietf.org/html/rfc3986#section-3.5. Of course, it's not forbidden to
> escape "/", so software that is interpreting a fragment identifier has to make
> sure it does the right thing.
>
> Regards, Martin.
>
>
>> Cheers,
>>
>>
>>>> +1 to Julian here -- there's no reason why non-ASCII chars need to be
>>>> percent-encoded when they occur inside a JSON document, only when they're in
>>>> a URI (or similar context).
>>>>
>>>> Cheers,
>>>>
>>>>

---

From: Julian Reschke <[email-masked]>
Date: Tue, 22 Nov 2011 14:06:49 +0100

> With this kind of dual-layer percent-encoding, we are deeply in quoting hell, and implementers will need all the help we can give them (and they will still get it wrong).

Indeed.

To add to the coverage, we should also consider whitespace and non-BMP 
characters:

JSON document:
{"Bjørn/Carsten/foo \uD834\uDD1E": "Fritz"}

JSON pointer:
"/Bjørn%2FCarsten%2Ffoo(X)(Y)"

(X) right now would be %20. Should it? Why escape it here already? (this 
applies to all characters that are currently disallowed expect '/' and '%')

(Y) imho should be the Unicode code point U+1D11E 
(<https://tools.ietf.org/html/rfc4627#section-2.5> has this example

So yes, the fact that a JSON name can be anything a JSON string can take 
is indeed a problem, because it doesn't leave us any characters as 
delimiters (so this is very different from XML vs XPath).

Best regards, Julian

---

From: Martin Thomson <[email-masked]>
Date: Wed, 23 Nov 2011 10:17:25 +1100

> So yes, the fact that a JSON name can be anything a JSON string can take is
> indeed a problem, because it doesn't leave us any characters as delimiters
> (so this is very different from XML vs XPath).

Maybe you could use a JSON string notation as your canonical form:

{ "Bjørn/Carsten/foo \uD834\uDD1E" : "Fritz" }
->
"/\"Bjørn/Carsten/foo \uD834\uDD1E\""

---

From: Julian Reschke <[email-masked]>
Date: Wed, 23 Nov 2011 10:12:26 +0100

> "/\"Bjørn/Carsten/foo \uD834\uDD1E\""

I like that.

Best regards, Julian

---

From: Julian Reschke <[email-masked]>
Date: Wed, 23 Nov 2011 14:17:43 +0100

Summarizing the open questions:

1) Which characters to *allow* in a JSON pointer (independently of where 
it appears)

1a) Allow non-ASCII (also, how to represent non-BMP characters)

1b) Allow ASCII characters that are reserved in URIs

2) What to use as delimiter, given the fact JSON identifiers allow 
anything that can occur in a JSON string (or: how to represent the "/")

3) How to map a JSON pointer to something that can be used inside a URI 
(path, query?, fragment identifier)

My two cents:

1a) yes, and represent a surrogate pair by the Unicode character it 
represents

1b) yes

2) I have no good idea; James suggested the Unicode replacement 
character, another idea would be a private use code point; using a 
JSON-escape ("\") has the drawback that it needs additional escaping in 
a URI, and that it may be hard to distinguish from the escaped variant 
when it appears in a JSON document, such as JSON-Patch)

3) encode-as-UTF8-then-percent-escape-everything-reserved-in-URI

Best regards, Julian

---

From: "Paul C. Bryan" <[email-masked]>
Date: Wed, 23 Nov 2011 10:12:06 -0800

So far I can't say I like it very much. Is the suggestion that every
member name in a JSON pointer reference must contain \" ... \"? This
makes for one ugly pointer in JSON, and absolutely guarantees that a
fragment identifier will contain multiple percent-encoded values, even
if there are no members that contain path delimiters.

#/a/b/c ← intuitive
#/%5C%22a%5C%22/%5C%22b%5C%22/%5C%22c%5C%22 ← not!  

Paul

---

From: Carsten Bormann <[email-masked]>
Date: Wed, 23 Nov 2011 22:37:26 +0100

> #/%5C%22a%5C%22/%5C%22b%5C%22/%5C%22c%5C%22 ← not!  

As usual, the guiding principle is that easy things should be easy, and that hard things should be possible.
Complicating the usual case (the only one that many people seem to have thought about yet) is indeed a big no-no.

Grüße, Carsten

---

From: Martin Thomson <[email-masked]>
Date: Thu, 24 Nov 2011 12:59:18 +1100

> Complicating the usual case (the only one that many people seem to have thought about yet) is indeed a big no-no.

A reasonable suggestion.  Maybe just drop the quotes and force an
escape for / instead of ":

  /Bjørn\/Carsten\/foo \uD834\uDD1E

---

From: "Martin J. Dürst" <[email-masked]>
Date: Thu, 24 Nov 2011 16:42:58 +0900

On 2011/11/24 10:59, Martin Thomson wrote:
> On 24 November 2011 08:37, Carsten Bormann<[email-masked]>  wrote:

For the moment, cutting off the surrogate pair at the end, and adding 
something to keep the space visible:

     /Bjørn\/Carsten\/foo bar

Yes indeed. So when defining json pointers, use '\/' to escape a '/' 
(and use '\\' to escape '\') that's part of a string rather than a 
hierarchy delimiter:

     /Bjørn\/Carsten\/foo bar

Then when using that as a fragment identifier, you have to escape the 
'\' (not allowed in URIs/IRIs), and of course the space, but not the '/':

     #/Bjørn%5C/Carsten%5C/foo%20bar

Now I really have no clue why surrogate pairs have to be escaped. Why 
not just use the character? This one didn't show in my mailer or 
browser, but one would assume that mostly the ones that actually show, 
at least in some places, will get used. So the actual pointer would be:

     /Bjørn\/Carsten\/foo 𝄞

In a fragment (IRI), that would look like:

     #/Bjørn%5C/Carsten%5C/foo%20𝄞

If an URI fragment is needed, use UTF-8 and %-encoding:

     #/Bj%C3%B8rn%5C/Carsten%5C/foo%20%F0%9D%84%9E

Regards,    Martin.

---

From: "Paul C. Bryan" <[email-masked]>
Date: Thu, 24 Nov 2011 09:18:28 -0800

I don't think this will work because every JSON parser I've used so far
processes "\/" as solidus "/". By the time it's parsed, there is no
escaping left. Or did I misinterpret, and in actual fact the backslash
would be doubled in actual JSON expression (e.g. "/Bjørn\\/Carsten\\/foo
bar")? That could work. 

Paul