Subject: Re: [apps-discuss] Kathleen Moriarty's Discuss on draft-ietf-appsawg-json-merge-patch-06: (with DISCUSS)

From: James M Snell <[email-masked]>
Date: Mon, 18 Aug 2014 14:02:09 -0700

In the case of using Merge-Patch with the HTTP Patch method,
"validation" would typically occur using the Conditional HTTP Request
Mechanisms (ETags and If-Match/If-None-Match). These aren't fool
proof, of course, but they provide a Good Enough approximation. If
someone needed something more rigorous, they could layer it in.

- James

On Mon, Aug 18, 2014 at 1:58 PM, Kathleen Moriarty
<[email-masked]> wrote:
> Kathleen Moriarty has entered the following ballot position for
> draft-ietf-appsawg-json-merge-patch-06: Discuss
>
> When responding, please keep the subject line intact and reply to all
> email addresses included in the To and CC lines. (Feel free to cut this
> introductory paragraph, however.)
>
>
> Please refer to http://www.ietf.org/iesg/statement/discuss-criteria.html
> for more information about IESG DISCUSS and COMMENT positions.
>
>
> The document, along with other ballot positions, can be found here:
> http://datatracker.ietf.org/doc/draft-ietf-appsawg-json-merge-patch/
>
>
>
> ----------------------------------------------------------------------
> DISCUSS:
> ----------------------------------------------------------------------
>
> The draft looks very good and the security considerations in RFC5789
> cover most of the bases I would be concerned with.  I do have a question
> I'd like to discuss to see if it applies or not.
>
> Normally, in other spaces patches are validated and that might be as
> simple as making sure the hash of the patch provided by the source
> matches the value you calculated (not corrupted, *should* be from the
> source you think it's from).  I don't see any mention of this practice,
> is there a reason for that or should it be added?  Maybe it's not
> possible because of how the patch is delivered over HTTP, but I figured
> it was worth flagging to discuss quickly.
>
>
>
>

---

From: "Kathleen Moriarty" <[email-masked]>
Date: Mon, 18 Aug 2014 13:58:13 -0700

---

From: Paul Hoffman <[email-masked]>
Date: Wed, 20 Aug 2014 18:08:44 -0700

The patches described in this document can be delivered however the implementers want. The document explicitly shows how to do this for any MIME-aware transport. Some of those might come with hashes that the recipient are expected to check, but most of the common ones today are not. I don't think that advice about sending hashes, checking hashes, ignoring patches that don't come with hashes, and so on are any more applicable here than to any other formats that could cause changes in system settings.

--Paul Hoffman

---

From: Paul Hoffman <[email-masked]>
Date: Wed, 20 Aug 2014 18:50:42 -0700

On Aug 20, 2014, at 6:20 PM, Kathleen Moriarty <[email-masked]> wrote:

> On Wed, Aug 20, 2014 at 9:08 PM, Paul Hoffman <[email-masked]> wrote:
> Sure agreed, but it's common practice to do that sort of thing with patches, so mentioning the practice would be good since you are saying it's possible.  I think it could be helpful to implementers to understand that option.  Can you add a couple of sentences?

I'm not sure I can add those sentences in a useful fashion. With current delivery mechanisms (HTTP being the most common), MIME objects don't come with hashes for checking. So giving that advice would cause developers to look for something that they won't have, namely hashes of the MIME object they just got.

I do hope you're not asking us to change the format to include an internal hash. That would require inventing JSON canonicalization (which the JOSE WG has flailed miserably on) and say "canonicalize, pick a hash function, hash, and add that to the patch object". The recipient would need to pull the hash out before processing the patch, canonicalize the result, and check the hash. This is conceptually possible but it seems onerous, and has not been done in any of the earlier patch formats that have been standardized in the IETF.

--Paul Hoffman

---

From: James M Snell <[email-masked]>
Date: Wed, 20 Aug 2014 19:00:53 -0700

On Aug 20, 2014 6:50 PM, "Paul Hoffman" <[email-masked]> wrote:
[Snip]
>
> I'm not sure I can add those sentences in a useful fashion. With current
delivery mechanisms (HTTP being the most common), MIME objects don't come
with hashes for checking. So giving that advice would cause developers to
look for something that they won't have, namely hashes of the MIME object
they just got.

+1

> I do hope you're not asking us to change the format to include an
internal hash. That would require inventing JSON canonicalization (which
the JOSE WG has flailed miserably on) and say "canonicalize, pick a hash
function, hash, and add that to the patch object". The recipient would need
to pull the hash out before processing the patch, canonicalize the result,
and check the hash. This is conceptually possible but it seems onerous, and
has not been done in any of the earlier patch formats that have been
standardized in the IETF.

+1... Adding an internal hash would not work with this mechanism without
taking a major step backwards. I do not see the value.

- James

> --Paul Hoffman
> _______________________________________________
> apps-discuss mailing list
> [email-masked]
> https://www.ietf.org/mailman/listinfo/apps-discuss

---

From: Carsten Bormann <[email-masked]>
Date: Thu, 21 Aug 2014 07:19:13 +0100

> On 21.08.2014, at 02:50, Paul Hoffman <[email-masked]> wrote:
> 
> inventing JSON canonicalization

If that ever happens, it is easy to define another media type for validating hashes that carries a pair of a patch (e.g. In merge patch format) and a hash. (Or, a MAC...)

Grüße, Carsten

---

From: Nico Williams <[email-masked]>
Date: Thu, 21 Aug 2014 01:30:45 -0500

My take is that this patch is not as general as RFC6902, but for some
use-cases it is easier to construct a patch of this form, therefore it
has utility, and depending on the edits one wishes to make, one of the
two patches can be more verbose than the other.

(E.g., making changes to various random parts of a large JSON text is
likely to produce a smaller patch with RFC6902, but some template-type
edits will be easier with the new patch.)

Still, if I had to pick just one JSON patch format, I'd pick a
path-based one like RFC6902.

Nico
--

---

From: Carsten Bormann <[email-masked]>
Date: Thu, 21 Aug 2014 09:52:26 +0100

This format has a sweet spot, which is:

O replacement points are values of map members, only
O null never occurs as a value of a map member to be emplaced or is otherwise innocuous
O replacement points are not deeply nested

RFC 6902 obviously is more general.  Merge-patch captures and formalizes what many people already have been doing (often mistakenly using PUT with the original media type, not PATCH).

Grüße, Carsten

---

From: Paul Hoffman <[email-masked]>
Date: Thu, 21 Aug 2014 07:11:17 -0700

Pete Resnick hates it when I just roll over on DISCUSSes that I think will make the document a bit worse, so I'll do another round on this, then probably roll over. Your call.

On Aug 20, 2014, at 7:14 PM, Kathleen Moriarty <[email-masked]> wrote:

> Not to worry, I wasn't asking for anything to be added other than text in the security considerations for when it was possible, keying off of your first response, "The document explicitly shows how to do this for any MIME-aware transport. Some of those might come with hashes that the recipient are expected to check, but most of the common ones today are not."  Obviously, it would not be something required since it doesn't always get delivered, but I don't see a reason to exclude mention of this practice when it provides additional integrity checking when it is available.

There is a very good reason not to: it will likely cause false negative rejections much more often than catching positives. JSON is a text format, and this merge patch format creates patches small enough to easily be sent around in email. Email clients tend to add or strip trailing whitespace and change line ending characters; so does FTP. Thus, a hash will likely say "this is bad" even when it is perfectly good. Thus, encouraging the use of hash checking at all is causing more operational security errors.

> I guess I find it odd that this has not been done before, so I'm guessing there has not been security issues yet that caused an update in practices?

Correct. For JSON or XML, sending a hash and expecting systems to reject messages because of text conversions is a bad practice. The best practice of "check the hash on this patch" is quite appropriate for .tgz .zip tarballs, and that's where you see it in the security community.

--Paul Hoffman

---

From: John C Klensin <[email-masked]>
Date: Thu, 21 Aug 2014 10:59:56 -0400

--On Thursday, August 21, 2014 07:11 -0700 Paul Hoffman
<[email-masked]> wrote:

>...
> There is a very good reason not to: it will likely cause false
> negative rejections much more often than catching positives.
> JSON is a text format, and this merge patch format creates
> patches small enough to easily be sent around in email. Email
> clients tend to add or strip trailing whitespace and change
> line ending characters; so does FTP. Thus, a hash will likely
> say "this is bad" even when it is perfectly good. Thus,
> encouraging the use of hash checking at all is causing more
> operational security errors.
>...

Paul, 

I agree with your analysis and the conclusion that, Kathleen's
suggested change would make the document worse and, unless
people were smart enough to ignore it, increase false negatives.
That could be fixed by a lot more text, but I don't think that
would be worth it.

However, a nit and a more general comment:

Nit: With email clients, "adjusting" trailing white space and
line ending is long-established and normal behavior to adapt
textual messages to the local environment.  That behavior cannot
be in violation of IETF Standards because we have never (IMO,
wisely) specified the behavior of MUAs.  However, barring local
extensions, FTP allows only two transfer encoding types for
stream-type data that are still relevant in the contemporary
Internet.  If "type ascii" is used, a canonical form is
specified and an FTP client or server are in violation of the
spec if anything else is sent. Of course, one or the other might
convert between that form and local conventions -- that was
actually the intent and may be what you were referring to.
"Type ASCII" is implausible (bad things will happen) for, e.g.,
ISO 8859 or non-ASCII-repertoire Unicode data.  If "type image"
is used, whatever exists on the sender side is expected to be
delivered, bit-by-bit intact, to the receiver; anything else is
a protocol violation.  

BTW, if anyone cares about an analogy for "type ASCII" for
general Unicode data or things that can be mapped into it, I've
tried to float a "TYPE U" proposal several times but it has
gotten no traction in the past.

General comment, more to Kathleen and the IESG than to you:
There are many formats that are essentially plain text at the
character level, including JSON and XML as well as ordinary
email.  When hashes (or other types of signatures) are being
computed over them, we should try to be a lot more cautious than
we have been about the importance of specifying a canonical form
and computing the hash over it.  FWIW, the problem gets a _lot_
worse when we move past ASCII data and into Unicode and its
various canonical and encoding forms.  That caution could take
the form of warning about a lot of false negatives and not
overreacting to them or could actually involve a canonical form
specification.  But, IMO, we have been somewhat too willing to
ignore the problem or pretend it isn't there.

   john

---

From: Kathleen Moriarty <[email-masked]>
Date: Wed, 20 Aug 2014 21:20:05 -0400

On Wed, Aug 20, 2014 at 9:08 PM, Paul Hoffman <[email-masked]> wrote:

> The patches described in this document can be delivered however the
> implementers want. The document explicitly shows how to do this for any
> MIME-aware transport. Some of those might come with hashes that the
> recipient are expected to check, but most of the common ones today are not.
> I don't think that advice about sending hashes, checking hashes, ignoring
> patches that don't come with hashes, and so on are any more applicable here
> than to any other formats that could cause changes in system settings.
>
> Sure agreed, but it's common practice to do that sort of thing with
patches, so mentioning the practice would be good since you are saying it's
possible.  I think it could be helpful to implementers to understand that
option.  Can you add a couple of sentences?

Thanks.

-- 

Best regards,
Kathleen

---

From: Kathleen Moriarty <[email-masked]>
Date: Wed, 20 Aug 2014 22:14:30 -0400

> taking a major step backwards. I do not see the value.
Not to worry, I wasn't asking for anything to be added other than text in
the security considerations for when it was possible, keying off of your
first response, "The document explicitly shows how to do this for any
MIME-aware transport. Some of those might come with hashes that the
recipient are expected to check, but most of the common ones today are
not."  Obviously, it would not be something required since it doesn't
always get delivered, but I don't see a reason to exclude mention of this
practice when it provides additional integrity checking when it is
available.

I guess I find it odd that this has not been done before, so I'm guessing
there has not been security issues yet that caused an update in practices?

Thanks.

-- 

Best regards,
Kathleen

---

From: Kathleen Moriarty <[email-masked]>
Date: Thu, 21 Aug 2014 10:20:04 -0400

On Thu, Aug 21, 2014 at 10:11 AM, Paul Hoffman <[email-masked]>
wrote:

> Pete Resnick hates it when I just roll over on DISCUSSes that I think will
> make the document a bit worse, so I'll do another round on this, then
> probably roll over. Your call.
>
Just looking for a reasonable explanation and appreciate the discussion.  I
don't want to see a change that doesn't make sense either and am happy to
see a concern is unfounded if that's the case.  I'd just rather make sure
we address security concerns when needed.

> > Not to worry, I wasn't asking for anything to be added other than text
> in the security considerations for when it was possible, keying off of your
> There is a very good reason not to: it will likely cause false negative
> rejections much more often than catching positives. JSON is a text format,
> and this merge patch format creates patches small enough to easily be sent
> around in email. Email clients tend to add or strip trailing whitespace and
> change line ending characters; so does FTP. Thus, a hash will likely say
> "this is bad" even when it is perfectly good. Thus, encouraging the use of
> hash checking at all is causing more operational security errors.
>

This is very helpful and a good explanation as to why this should not be
done, thank you.

>
> > I guess I find it odd that this has not been done before, so I'm
> guessing there has not been security issues yet that caused an update in
> practices?
>
> Correct. For JSON or XML, sending a hash and expecting systems to reject
> messages because of text conversions is a bad practice. The best practice
> of "check the hash on this patch" is quite appropriate for .tgz .zip
> tarballs, and that's where you see it in the security community.
>

Yes, and it's also used when downloading source code (may be in those
formats) to verify the integrity and source (assuming the provided hash
wasn't altered too).  I don't think I've seen a mismatch where code has
been replaced by an attacker for a while, but that's probably because this
is commonplace to check and source providers check their servers and what
they provide hasn't been altered on regular intervals.

I'll drop the discuss and hopefully this won't be an issue in this area or
it will get fixed in the future.

Thanks,
Kathleen

-- 

Best regards,
Kathleen