Subject: [core] explicit option for URI query part

From: Peter Bigot <[email-masked]>
Date: Mon, 19 Jul 2010 17:32:43 -0500

Technically, the query part of a URI is distinct from the path part, but it
seems that coap-01 currently requires that they be combined in the Uri-Path
option for functions like resource description query filtering.  Suggest a
distinct option Uri-Query be added, which if absent indicates lack of a
query part, and if present comprises the query part exclusive of the
introductory hook ("?").

This would restore the distinction between the path and query portions of a
URI, and make it easier for an end-point to detect the presence of a query
part.  And make it easier to add fragment support (as another option) if
that should ever be useful.

Peter

---

From: Carsten Bormann <[email-masked]>
Date: Tue, 20 Jul 2010 14:02:28 +0200

[Apologies for sending duplicates; there is an IPv6 problem between two mail servers waiting to be debugged...
My real mail address is [email-masked].]

On Jul 20, 2010, at 00:32, Peter Bigot wrote:

> Uri-Query

Interesting.
That would be a critical option, so a node that does not want to implement queries would reject them without any additional code.

The split into scheme/authority/path could make simple nodes simpler, because they only have to implement Uri-Path.
On the other hand, accepting this implementation strategy means that nodes should not send gratuitous Scheme/Authority options if the default value is fine.
(This is probably already a good idea for space reasons.  But it does mean a little more code for nodes that do need to send them sometimes.)

Note that this would mean a quite different default URI parse strategy than for HTTP, where sending a Host: header is mandatory.
With a URI of
	coap://ns.tzi.org:61616

I certainly don't have to send the Uri-Scheme option (as the default value is fine), but do I have to send the Uri-Authority header?
If we support what is called "virtual hosts" in HTTP (multiple authority values on one IP address), one would have to.

How important are virtual hosts then on the CoAP side?
(Less important on IPv6 than they were for HTTP on IPv4, but how much less?)

Gruesse, Carsten

---

From: "Angelo P. Castellani" <[email-masked]>
Date: Tue, 20 Jul 2010 15:31:56 +0200

Splitting query parameters from URI is a good idea in my opinion,
however for parsing simplicity we can also do something more.

A Query-Parameter option already splitted up in the different parameters.

Example: "parm1=hello&parm2=world"

This would became two Query-Parameter options respectively with value
"parm1=hello" and "parm2=world"

Even better if we can figure out a way to split also parameter name
from the value, but I can't came up with any solution for this...

About Uri-Authority I think is a very important feature for
compatibility with HTTP, however why isn't that option called Host
(like in HTTP?) and defined to carry the same value as in HTTP?

Angelo

On Tue, Jul 20, 2010 at 14:02, Carsten Bormann <[email-masked]> wrote:
> _______________________________________________
> core mailing list
> [email-masked]
> https://www.ietf.org/mailman/listinfo/core
>

---

From: Peter Bigot <[email-masked]>
Date: Tue, 20 Jul 2010 09:27:31 -0500

Re "Query-Parameter": we could do that, but....

coap-01 section 2.5.2 talks about splitting the URI into "its three parts";
where I was going is that there are actually five major parts to a URI: the
scheme, the authority and path, the query, and the fragment.  Most of the
time we only care about the path.  (The text describing how the full URI can
be created by concatenating the parts or their defaults needs to be updated
to add the missing glue text: the suffix ":" on the scheme, the prefix "//"
on the authority, the prefix "/" on the path, etc.)

My reluctance to go further and create a Query-Parameter option is that the
common syntax of a query being ampersand-separated key=value pairs is not
syntactically part of the URI specification (RFC3986), not even for HTTP
URIs.  Though as long as the option name doesn't include "Uri-", maybe it
would be OK to add that and derive the default value for Uri-Query from it.
It would certainly be convenient.

An argument could be made that we should have a Host option that behaves as
you suggest, but it is not the same thing as Uri-Authority, neither
syntactically (authority allows an optional userinfo "@") nor semantically.
It's also unnecessary: HTTP requires the Host header keyword because it has
stripped that information from the URI that appears in the request-line.
With the full set of URI options, CoAP doesn't need to do that, and
translations to/from HTTP's Host header can use the Uri-Authority option
without needing to impact its name (and its use for other schemes that don't
have HTTP's limitations).

HTTP needs the Host header because the authority part of the URI is
otherwise present only in the envelope (the TCP packet).  Similarly, the
Uri-Authority provides an explicit representation of what is otherwise the
end-point address.  My initial feeling is that I don't want Uri-Authority to
be a required option because (a) it can be pretty long, and (b) the only
things that will care about it are proxies and (as Carsten points out) any
CoAP end-point that presents a virtual host interface.  On the other hand,
I'm concerned about having the default value of Uri-Authority be the empty
string: that means something different than an actual host+port as
determined by the endpoint, and it may not be reasonable to require the
implementation to provide envelope information so that the correct default
value can be reconstructed.  Making Uri-Authority required would eliminate
propagating envelope information, as well as the current requirement that a
client needs to know whether it's talking to a proxy in order to determine
whether it needs to provide the Uri-Authority option.  I'll tell you though:
I expect my end-points to ignore it.

If we do start adding all these options, including ones like Query-Parameter
that could appear multiple times, we're going to need a mechanism to support
more than 15 options in a header.

Peter

On Tue, Jul 20, 2010 at 8:31 AM, Angelo P. Castellani <[email-masked]
> wrote:

> > That would be a critical option, so a node that does not want to
> implement queries would reject them without any additional code.
> >
> > The split into scheme/authority/path could make simple nodes simpler,
> because they only have to implement Uri-Path.
> > On the other hand, accepting this implementation strategy means that
> nodes should not send gratuitous Scheme/Authority options if the default
> value is fine.
> > (This is probably already a good idea for space reasons.  But it does
> mean a little more code for nodes that do need to send them sometimes.)
> >
> > Note that this would mean a quite different default URI parse strategy
> than for HTTP, where sending a Host: header is mandatory.
> > With a URI of
> >        coap://ns.tzi.org:61616
> >
> > I certainly don't have to send the Uri-Scheme option (as the default
> value is fine), but do I have to send the Uri-Authority header?
> > If we support what is called "virtual hosts" in HTTP (multiple authority
> values on one IP address), one would have to.
> >

---

From: Carsten Bormann <[email-masked]>
Date: Tue, 20 Jul 2010 17:31:10 +0200

Peter,

+1 for most of what you said.

Two comments though:

1) always requiring the Uri-Authority option would bloat packets badly (and, as you say, simple nodes will ignore that bloat anyway).
I don't see what's wrong with keeping the "default/empty authority" semantics as "the node that acts as server".
However, we need to either explicitly indicate virtual hosts in a URI or outlaw them to enable this.

2) Re:

On Jul 20, 2010, at 16:27, Peter Bigot wrote:

> we're going to need a mechanism to support more than 15 options in a header.

More than 15 options in a single CoAP message makes me shudder.
(It is very easy to add such a capability, even in a way that does not punish messages with less than 15 options.  But I'm not seeing the application.)

A more radical question:
Who needs lots and lots of queries?
I'm not seeing them very much in modern HTTP applications.

Gruesse, Carsten
(Still posting from Gmail while we wait for att.net to heal.)

---

From: Peter Bigot <[email-masked]>
Date: Tue, 20 Jul 2010 11:14:35 -0500

I don't want or expect to see dozens of options either, but I do think as a
protocol we should be prepared to accommodate them for situations we can't
foresee.  It will be more important if we have options that can be repeated
(as with a Query-Parameter), or if we start having so many optional options
that we need to use the fence-post mechanism to adjust for large
option-delta values (since each fencepost use contributes to the option
count).

With the current set I can see having Max-Age, Accepts, Uri-Scheme,
Uri-Authority, Uri-Path, and Etag all being necessary for a request directed
to a caching end-point.  That's using close to half the available spaces,
and we're still proposing new, presumably useful, options.

I think any such extension has to be formed as a critical option for it to
be useful.  Perhaps option type 15 as Option-Block, with its value being a
sequence of delta-length-value--encoded options.  Allow it to be repeated
for perverse situations.

Peter

On Tue, Jul 20, 2010 at 10:31 AM, Carsten Bormann <[email-masked]>wrote:

> (It is very easy to add such a capability, even in a way that does not
> punish messages with less than 15 options.  But I'm not seeing the
> application.)
>
> A more radical question:

---

From: Peter Bigot <[email-masked]>
Date: Tue, 20 Jul 2010 11:46:32 -0500

On Tue, Jul 20, 2010 at 10:31 AM, Carsten Bormann <[email-masked]>wrote:

> 1) always requiring the Uri-Authority option would bloat packets badly
> (and, as you say, simple nodes will ignore that bloat anyway).
> I don't see what's wrong with keeping the "default/empty authority"
> semantics as "the node that acts as server".
>

I don't think that semantics is currently explicit, and whether it's
workable depends on its interpretation.

Does it imply that, when reconstructing the URI from the options, it's
necessary to extract the envelope's source end-point address+host or
destination end-point address+host so that the server node can be explicitly
identified?  This would get really ugly, since "server" depends on the
method and which end is processing the message,  Plus, what happens if the
end-point received the message over multicast (which can't be the end-point
address because it can't be used for a UDP source address)?  Then the
reconstructed URI is different on each recipient.  That would introduce an
issue with recognizing an asynchronous response.

I propose making Uri-Authority have no default, rather than an empty string
default: an empty string doesn't conform to the ABNF for authority in
section 3.2 of RFC3986.  I would interpret "absent authority" as "no
authority specified", with the meaning of that delegated to standard
interpretation of a URI that lacks a "//" authority sequence.  If a
non-absent Uri-Authority is present, then the reconstructed URI includes the
"//" prefix followed by the option's value.

(This may be pragmatically equivalent to what you suggested.)

However, we need to either explicitly indicate virtual hosts in a URI or
> outlaw them to enable this.
>

If that's true (not convinced), outlaw 'em.  Life's complicated enough
already.  If you want HTTP, use it.

Peter

---

From: Carsten Bormann <[email-masked]>
Date: Tue, 20 Jul 2010 19:14:27 +0200

On Jul 20, 2010, at 18:14, Peter Bigot wrote:

> Option-Block, with its value being a sequence of delta-length-value--encoded options.

Yep, I had that in an early version of coap-misc, and I took it out because I don't think we should plan for a bazillion options.
But yes, if we move toward micro-options like the Uri-Query element, we could go back to that.
The "Options" option of coap-misc was a single option that was alternative to having other options at the top level.
Like this (ignore the idea to limit to 7 options in a message; we now have up to 15 just from the header).

Options Option
--------------

There is no hard boundary that can be given for the number of options
required in a message.  While messages will rarely require more than 4
or 5 options, some will, and it would be wasteful to provide a large
enough option counter in the message header to cater for the worst
case.

Instead, we propose to limit the option counter in the message header
to a maximum value of 7 (i.e., it can be represented in 3 bits).
If 8 or more options are required, **all** options are packaged into an
`Options` option, giving explicitly the length of the option part of
the message.  This option, if present, must be the first and only
option (so the option counter in the message header will be 1).
The Options option MAY NOT be used unless needed (i.e., 8 or more
options in the message).  As a consequence, Options options never can nest.

Within the Options option, the running option number increased by the
options delta starts at zero again.

Gruesse, Carsten

---

From: Carsten Bormann <[email-masked]>
Date: Tue, 20 Jul 2010 19:24:39 +0200

On Jul 20, 2010, at 18:46, Peter Bigot wrote:

> On Tue, Jul 20, 2010 at 10:31 AM, Carsten Bormann <[email-masked]> wrote:
> I don't think that semantics is currently explicit,

Let's fix that then.

> and whether it's workable depends on its interpretation.

Let's do a workable interpretation then :-)

> Does it imply that, when reconstructing the URI from the options, it's necessary to extract the envelope's source end-point address+host or destination end-point address+host so that the server node can be explicitly identified?  

No, the idea is that there is an identifiable receiver of the request/emitter of the response, and that is addressed by the request/emits the response.

> This would get really ugly, since "server" depends on the method and which end is processing the message,  Plus, what happens if the end-point received the message over multicast (which can't be the end-point address because it can't be used for a UDP source address)?  

Indeed, if a multicast request is to be useful in any way, it needs to *leave out* authority (or have a group authority), and make the authority actually acting on the request implicitly derived.

> Then the reconstructed URI is different on each recipient.  That would introduce an issue with recognizing an asynchronous response.

Not so sure about that.  If f::0:0 tells me their /bar is 14, that may be exactly what I want to know.

> I propose making Uri-Authority have no default, rather than an empty string default: an empty string doesn't conform to the ABNF for authority in section 3.2 of RFC3986.  

Really?  Page 18:

      host        = IP-literal / IPv4address / reg-name

Page 21:

      reg-name    = *( unreserved / pct-encoded / sub-delims )

Note that *(...) means zero or more of.  It goes on to say:

   If the URI scheme defines a default for host, then that default
   applies when the host subcomponent is undefined or when the
   registered name is empty (zero length).  For example, the "file" URI
   scheme is defined so that no authority, an empty host, and
   "localhost" all mean the end-user's machine, whereas the "http"
   scheme considers a missing authority or empty host invalid.

> If that's true (not convinced), outlaw 'em.  Life's complicated enough already.  If you want HTTP, use it.

Sounds good to me.

Gruesse, Carsten

---

From: Carsten Bormann <[email-masked]>
Date: Tue, 20 Jul 2010 09:35:13 +0200

Interesting.

Gruesse, Carsten