Subject: Re: [apps-discuss] +exi

From: Peter Saint-Andre <[email-masked]>
Date: Tue, 13 Dec 2011 10:45:19 -0700

Restarting this thread to get closure...

On 11/15/11 8:21 PM, Zach Shelby wrote:
> On Nov 16, 2011, at 11:06 AM, Bjoern Hoehrmann wrote:
> 
>> * Peter Saint-Andre wrote:
>>> So, let's say I have "foo" data that can be represented in either
>>> plain old XML or as EXI. If I send that "foo" data as plain old
>>> XML, the "application/foo+xml" media type is right. If I send
>>> that "foo" data as EXI, the "application/foo+exi" media type
>>> seems wrong, and so does the "application/exi" media type (just
>>> as "application/xml" would not be right for the first encoding).
>>> Sure, I don't particularly want to duplicate all "+xml" entries
>>> with "+exi" entries, but I don't think that every protocol or
>>> community that sends around XML data will also send around
>>> EXI-encoded data.
>> 
>> The idea is that "EXI" is more like "gzip", so with HTTP you would
>> do
>> 
>> Content-Type: application/foo+xml Content-Encoding: exi
>> 
>> As you would for gzip-compressed content.
> 
> Ouch, don't do that.
> 
> The main interest right now for EXI is in constrained networks, and
> in particular over CoAP. We don't have support for indicating
> content-encoding in CoAP. Besides, the current media type for EXI is
> application/exi. Thus it would make perfect sense to have +exi
> entries. Furthermore, EXI is totally different than gzip, which is a
> generic encoding that can be applied to anything. EXI can be applied
> only to XML objects, and in particular EXI is often used in a schema
> mode where it is only applicable to a single schema. Thus the form
> application/schema+exi makes even more sense, as that EXI format is
> actually specific to that particular schema and the media type tells
> you everything that is needed to decode the representation.

I chatted with a few folks about this in Taipei and afterward.

As I understand it (correct me if I'm wrong), EXI has two modes:

1. Straight compression of the text representation of XML (text in,
EXI-compressed data out). To make use of the data, you then decompress
it back into the angle-brackety text representation of XML that we all
know and love.

2. Direct, binary representation of the XML infoset. In this case, you
never have the data in a angle-brackety text representation, instead it
is always a binary representation.

It seems to me that #1 is similar to gzip, i.e., the textual
representation is encoded using EXI compression, so we'd have this:

   Content-Type: application/foo+xml
   Content-Encoding: exi

It seems to me that #2 might not be an encoding of the relevant +xml
content type, but might be a different content type, so +exi might be
appropriate.

Peter

-- 
Peter Saint-Andre
https://stpeter.im/

---

From: SM <[email-masked]>
Date: Tue, 13 Dec 2011 10:02:33 -0800

>content type, but might be a different content type, so +exi might be

I don't know whether Item B at 
http://www.ietf.org/proceedings/78/minutes/yam.txt is relevant to the 
discussion.

Regards,
-sm

---

From: Peter Saint-Andre <[email-masked]>
Date: Tue, 13 Dec 2011 14:19:30 -0700

> http://www.ietf.org/proceedings/78/minutes/yam.txt is relevant to the

Sort of. :)

The case of gzipped SVG is a good example of an encoding of a content
type. The case of a binary representation of an XML infoset might be an
example of a content type (as opposed to a compressed encoding of
textual XML, which I think we all agree is an encoding of a content
type). Before IETF 82, I was not aware of the EXI mode that represents
an XML infoset directly in binary, but some folks there enlightened me,
so I think there is more subtlety here than was reflected in this thread
in early November.

Peter

-- 
Peter Saint-Andre
https://stpeter.im/

---

From: Julian Reschke <[email-masked]>
Date: Tue, 13 Dec 2011 22:40:27 +0100

> The case of gzipped SVG is a good example of an encoding of a content

Well, until people start looking for a content type to map their "svgz" 
files to, or need to embed gzipped SVG into a data URI.

The decision to clarify that "image/svg+xml" isn't useable for that was 
right, but this use case may really required a separate content type.

> so I think there is more subtlety here than was reflected in this thread

I'm not sure that I understand the difference. Both are compression 
formats. One may be specialized for XML, but otherwise?

Best regards, Julian

---

From: Carine Bournez <[email-masked]>
Date: Tue, 13 Dec 2011 16:58:16 -0500

> is always a binary representation.

It is not exactly 2 modes. The EXI 1.0 Recommendation only specifies a 
compressed encoding of an XML Infoset. Applications that include an EXI 
processor may use different approaches around the encoding/decoding process. 
So, you can have applications with a (text) XML document as an input
(what you describe as #1), and applications that never use a text
serialization (your #2).

> content type, but might be a different content type, so +exi might be

The W3C EXI Working Group has decided not to follow the +exi path, 
since it involved a potentially infinite number of types to register.
There could also be conformance issues: an EXI stream not encoded
from a text version might not be necessarily serialized as a conformant
"foo" after decoding from its EXI form. It needs some specification 
work for each foo+exi.
The generic application/exi content-type is meant to be used for cases 
where there is no serialization from/to a standard format (then no 
content-type can be used), as well as for protocols that have no 
content-encoding capability.

-- 
Carine Bournez -+- W3C Europe

---

From: Peter Saint-Andre <[email-masked]>
Date: Tue, 13 Dec 2011 15:36:11 -0700

> content-encoding capability.

Thank you for the clarifications.

The example that raised this issue for me was the ZigBee Alliance Smart
Energy Profile 2.0, which supports both a text serialization of their
XML format and an EXI representation. Tom Herbst and Paul Duffy can
describe that use case more accurately than I can, but based on what
you've said it seems that they would use application/sep+xml and note a
content encoding of EXI for the second case. However, it is possible
that (if their transport/transfer protocols do not have a way to note
the content encoding) they might use application/exi for the second
case. I'll let Tom and Paul speak to that. Zach Shelby might also have
opinions in the matter. In any case, I think they have the information
they need to make the right decision in the ZigBee Alliance (and if not
they can speak up).

Peter

-- 
Peter Saint-Andre
https://stpeter.im/

---

From: Mark Nottingham <[email-masked]>
Date: Wed, 14 Dec 2011 09:37:27 +1100

> (what you describe as #1), and applications that never use a text

If I remember correctly, it also has modes where the schema (as a separate document) is required to be able to decode the document; is that still the case, Carine?

Cheers,

--
Mark Nottingham   http://www.mnot.net/

---

From: Carine Bournez <[email-masked]>
Date: Wed, 14 Dec 2011 04:23:36 -0500

> If I remember correctly, it also has modes where the schema (as a separate document) is required to be able to decode the document; is that still the case, Carine?

Hi,
In the schema-informed mode the encoder and the decoder use a schema,
the EXI 1.0 version specifies XML schema informed grammars. The reference
to that schema is carried in the EXI header. The format of that "schemaId" 
is not specified for 1.0 processors, it can be whatever is practical for 
the application. It is *optional* in the EXI header, even when a schema is
in use: applications can also "share" this information out of band. 
EXI 1.0 is intentionally liberal on a number of details and defers them to 
the application level, so that it can support a large range of use cases.

---

From: Paul Duffy <[email-masked]>
Date: Tue, 13 Dec 2011 16:58:55 -0600

Peter accurately described SEP2's intentions re: EXI usage.  We're 
essentially marshaling/un-marshaling an  internal infoset either...

1. direct to XML/text

2. direct to an EXI binary encoding.  Note these nodes never manipulate 
XML text.  This approach is particularly important for resource 
constrained nodes (i.e. memory not available to blow the EXI up into XML).

We tend to look at these as two different encodings, not one as a 
compression of the other.

---

From: Thomas Herbst <[email-masked]>
Date: Tue, 13 Dec 2011 15:00:12 -0800

Also, we are trying to be as http/coap agnostic as possible.

---

From: Zach Shelby <[email-masked]>
Date: Fri, 16 Dec 2011 09:24:05 +0200

> the application level, so that it can support a large range of use cases.

Right. It is exactly these kinds of applications that are using schema informed mode natively (without intermediate XML), where the application/foo+exi media type helps to determine this information out of band. When registering foo+exi, the schema information should be included in the registration.  In addition to SE2 mentioned so far, also our specification on SenML [http://tools.ietf.org/html/draft-jennings-senml-07] is requesting an application/senml+exi media type. Both of these media types are aimed at constrained embedded devices.

Zach

-- 
Zach Shelby, Chief Nerd, Sensinode Ltd.
http://www.sensinode.com
http://zachshelby.org  - My blog "On the Internet of Things"
http://6lowpan.net - My book "6LoWPAN: The Wireless Embedded Internet"
Mobile: +358 40 7796297

---

From: Peter Saint-Andre <[email-masked]>
Date: Fri, 16 Dec 2011 09:15:46 -0700

On 12/16/11 12:24 AM, Zach Shelby wrote:
>> In the schema-informed mode the encoder and the decoder use a
>> schema, the EXI 1.0 version specifies XML schema informed grammars.
>> The reference to that schema is carried in the EXI header. The
>> format of that "schemaId" is not specified for 1.0 processors, it
>> can be whatever is practical for the application. It is *optional*
>> in the EXI header, even when a schema is in use: applications can
>> also "share" this information out of band. EXI 1.0 is intentionally
>> liberal on a number of details and defers them to the application
>> level, so that it can support a large range of use cases.
> 
> Right. It is exactly these kinds of applications that are using
> schema informed mode natively (without intermediate XML), where the
> application/foo+exi media type helps to determine this information
> out of band. When registering foo+exi, the schema information should
> be included in the registration.  In addition to SE2 mentioned so
> far, also our specification on SenML
> [http://tools.ietf.org/html/draft-jennings-senml-07] is requesting an
> application/senml+exi media type. Both of these media types are aimed
> at constrained embedded devices.

Zach, thanks for clarifying the matter. Perhaps it would make sense for
someone to write a brief I-D describing the rationale for media types
with the +exi suffix and the information required to register such types.

Peter

-- 
Peter Saint-Andre
https://stpeter.im/

---

From: Peter Saint-Andre <[email-masked]>
Date: Fri, 16 Dec 2011 12:11:14 -0700

> with the +exi suffix and the information required to register such types.

BTW, Section 4.2.8 of draft-freed-media-type-regs-01 is apropos:

https://datatracker.ietf.org/doc/draft-freed-media-type-regs/

Peter

-- 
Peter Saint-Andre
https://stpeter.im/

---

From: "Paul E. Jones" <[email-masked]>
Date: Fri, 16 Dec 2011 14:26:32 -0500

What might also be interesting to understand is how register +exi would be
different than registering other binary XML formats.  Would we need
+fastinfoset and +BiM and so on?

I'm not opposed to +exi if that helps in decoding the data somehow, but it
doesn't seem like it would add any more value than signaling the media type
and indicating the content encoding.  In fact, the latter might be used to
indicate various modes of EXI, if that was necessary.

Paul

> -----Original Message-----
> From: [email-masked] [mailto:apps-discuss-
> [email-masked]] On Behalf Of Peter Saint-Andre
> Sent: Friday, December 16, 2011 11:16 AM
> To: Zach Shelby
> Cc: [email-masked]; Mark Nottingham; Thomas Herbst; apps-
> [email-masked]
> Subject: Re: [apps-discuss] +exi
> 
> >> the EXI 1.0 version specifies XML schema informed grammars.
> >> The reference to that schema is carried in the EXI header. The format
> >> of that "schemaId" is not specified for 1.0 processors, it can be
> >> whatever is practical for the application. It is *optional* in the
> >> EXI header, even when a schema is in use: applications can also
> >> "share" this information out of band. EXI 1.0 is intentionally
> > Right. It is exactly these kinds of applications that are using schema
> > informed mode natively (without intermediate XML), where the
> > application/foo+exi media type helps to determine this information out
> > of band. When registering foo+exi, the schema information should be
> > included in the registration.  In addition to SE2 mentioned so far,
> > also our specification on SenML
> _______________________________________________
> apps-discuss mailing list
> [email-masked]
> https://www.ietf.org/mailman/listinfo/apps-discuss

---

From: Carine Bournez <[email-masked]>
Date: Sat, 17 Dec 2011 05:41:03 -0500

> Right. It is exactly these kinds of applications that are using schema informed mode natively (without intermediate XML), where the application/foo+exi media type helps to determine this informatio…

It seems that you could use application/foo and the content-encoding: exi
instead of registering a different media type. Is your application using a
protocol that has no content-encoding? or do you mean that you use a 
different schema for the exchange of a foo serialized in EXI than a 
text foo? A quick look at http://tools.ietf.org/html/draft-jennings-senml-07 
makes me think it is the latter, for SenML. It does not seem a very nice
way to convey the schema information, SchemaId is meant to serve this 
purpose, not the media type. Since you need to carry other EXI options,
out-of-band or in the EXI header, why not using SchemaId?.

---

From: Zach Shelby <[email-masked]>
Date: Sat, 17 Dec 2011 21:34:20 +0200

On Dec 17, 2011, at 12:41 PM, Carine Bournez wrote:

> protocol that has no content-encoding?

This is one aspect, these applications are often aimed at CoAP, where content-encodings are more difficult. 

> or do you mean that you use a 
> text foo?

Yes, this is entirely possible for some applications as strict schema informed mode has some more limitations when it comes to schemas. And since in these cases we work directly with EXI, you can't make the assumption that there is even an XML text mode foo. There may only be for example foo+json and foo+exi. 

> A quick look at http://tools.ietf.org/html/draft-jennings-senml-07 

I don't find the SchemaId all that useful. First of all, you need to invoke your EXI parser to even get at that. It is more useful to immediately look at the content-type to decide which parser to throw a representation at. A strictly defined foo+exi registration would tell you that nicely. 

Anyways, I think it is worthwhile to write a short draft on what the registration requirements for +exi would be as Peter suggests. I am happy to help with that.

Zach

---

From: Martin Thomson <[email-masked]>
Date: Sun, 18 Dec 2011 14:11:40 +1100

> I don't find the SchemaId all that useful. First of all, you need to invoke your EXI parser to even get at that. It is more useful to immediately look at the content-type to decide which parser to…

ISTM that a link relation type for schema would make some sort of
difference.  The JSON schema draft had a "describedby" relation type
that might fit the bill (though "schema" is shorter).