Subject: Re: [core] #4: URI Encoding

From: "Stuber, Michael" <[email-masked]>
Date: Fri, 11 Jun 2010 15:30:54 -0700

Aren't you going to have too much overhead with the symbol table?  

-----Original Message-----
From: Paul Duffy <[email-masked]>
Sent: Friday, June 11, 2010 3:08 PM
To: Angelo P. Castellani <[email-masked]>
Cc: core issue tracker <[email-masked]>; core <[email-masked]>
Subject: Re: [core] #4: URI Encoding

DEFLATE appears to be popping up quite frequently re: various 
constrained node capabilities.

Viable for URIs?

On 6/11/2010 10:52 AM, Angelo P. Castellani wrote:
> I've two different proposals on this topic:
>
> 1) Compressed URI using a dictionary:
>
> Exploiting the fact the URI is a 7-bit ASCII, we can represent most
> frequent URI characters (regex format: [0-9a-zA-Z/]) using a 6 bit
> encoding, a NULL (\x00) octet signals that an uncompressed character
> is following. Every 3 compressed chars we have 6 unused bits that can
> be used to represent another character (0.75 ratio if characters are
> all from the dictionary).
>
> Maybe some predefined dictionaries can be agreed, and the used
> dictionary can be chosen in a preceding field (another example:
> [a-zA-Z%+._/\|=&?-], etc.)
>
> Reference: http://www.ietf.org/mail-archive/web/core/current/msg00166.html
>
> 2) Binary-URI encoding:
>
> Binary-URI is a compressed mapping of the URI:
> * It is N bytes long, where N is the depth of the resource in the URI. example:
>
> (a) /document/letter/order.html DEPTH: 3
> (b) / DEPTH: 0
> (c) /index.html DEPTH: 1
>
> * byte K is equal to the sum modulo 256 of ASCII values forming the
> K-level resource name in the tree:
>
> (a) document = 95, letter = 144, order.html = 255. So Binary-URI = {
> 95, 144, 255 } (c-style representation)
> (b) no byte so Binary-URI = {}
> (c) index.html = 251. Binary-URI = { 251 }
>
> Here is the code of the program used to compute the values:
>
> sum.c
> #include<stdio.h>
> #include<stdlib.h>
>
> int main (int argc, char* argv[]) {
>         if (argc==2) {
>                 unsigned char* p = argv[1];
>                 int sum=0;
>                 do {
>                         sum+=*p;
>                 } while (*(++p));
>                 printf("word: \"%s\" sum: %d\n", argv[1], sum&0xff);
>         }
> }
>
> Resources created on a CoAP server should not collide with each-other,
> to achieve this when we try to register a new resource the CoAP server
> should check if it is colliding with other available resources and
> accept the request only if this condition is satisfied.
>
> Reference: http://www.ietf.org/mail-archive/web/core/current/msg00133.html
>
> In my opinion both are useful, because (1) is useful when a CoAP node
> wants to access a resource which cannot be represented in Binary-URI
> format (CoAP node ->  HTTP server on the internet); (2) can be used
> when a CoAP server (a node) provides only non-colliding resources and
> thanks to this its URI can be represented using the very short and
> hierarchical Binary-URI format.
>
> Best,
> Angelo
>
> On Fri, Jun 11, 2010 at 11:31, core issue tracker<[email-masked]>  wrote:
>    
>> #4: URI Encoding
>> --------------------------------+-------------------------------------------
>>   Reporter:  zach@…              |       Owner:  zach@…
>>      Type:  enhancement         |      Status:  new
>>   Priority:  major               |   Milestone:
>> Component:  coap                |     Version:
>>   Severity:  -                   |    Keywords:
>> --------------------------------+-------------------------------------------
>>   The working group needs to consider the encoding of URIs in order to save
>>   overhead. The WG needs to converge on an approach to integrate in coap-01.
>>
>>   The following approaches have been suggested so far:
>>
>>   - Encode the URI with 7-bit ASCII encoding.
>>   - Encode the URI with Base64 encoding (requires mapping table).
>>   - Encode the URI with Ascii85 encoding (works on blocks of 5 characters)
>>   - Define a custom ASCII encoding only for valid URI characters.
>>
>>   There is an obvious tradeoff of space savings vs. complexity in these
>>   approaches. At the minimum 7-bit ASCII encoding is doable.
>>
>>   Furthermore, the Uri-code option was dropped in favor of integrating
>>   "short" URIs into the normal URI string option. Here is one proposed
>>   approach from Carsten:
>>
>>   - We could do an ASCII form of at least 4000 or so using /.xy or /!xy
>>   style shortcuts (essentially encoding the "codedness" in the first ASCII
>>   character).
>>
>> --
>> Ticket URL:<https://wiki.tools.ietf.org/wg/core/trac/ticket/4>
>> core<http://tools.ietf.org/core/>
>>
>> _______________________________________________
>> core mailing list
>> [email-masked]
>> https://www.ietf.org/mailman/listinfo/core
>>
>>      
> _______________________________________________
> core mailing list
> [email-masked]
> https://www.ietf.org/mailman/listinfo/core
>
>    

_______________________________________________
core mailing list
[email-masked]
https://www.ietf.org/mailman/listinfo/core

---

From: Carsten Bormann <[email-masked]>
Date: Sat, 12 Jun 2010 01:26:12 +0200

> DEFLATE

Deflate (RFC 1951, as used in RFC 2616 in conjunction with RFC 1950) derives its relatively high compression of text and most binary files from the repetition of byte sequences (it is based on the LZ77 algorithm, see
http://en.wikipedia.org/wiki/LZ77_and_LZ78
for a reasonably short description).

There is very little repetition in most URI paths, so Deflate on its own does not help there.
With a predefined dictionary (such as the one we made in RFC 3485 for RFC 3320), it *might* be useful if we manage to hit the right strings.

However, there is a lot of strain on constrained nodes: The whole dictionary would have to be available in all nodes.
Even with a small dictionary (such as the priority 1 subset from RFC 3485) it also still requires a lot of code.

If we really want to save space in encoding URIs, a simple static Huffman code seems more appropriate.
http://en.wikipedia.org/wiki/Huffman_coding has a nice example in the boxes at the right.
(Trying to use 6 bits for [0-9A-Za-z/] is a not particularly efficient form of static Huffman code.)
Again, URIs are too short to benefit much from a dynamic Huffman code, so a static table in the standard would be it.

But I'm not at all convinced the gain is worth all the code and the complexity.

The idea of hashing the URI path components is interesting.  But it requires the client to know all resources on a server to decide whether there is a collision.  (Alternatively, if we rule out collisions, we arbitrarily constrain the URI space, which may not work too well with user-generated names in URIs.)  Due to the birthday paradox (http://en.wikipedia.org/wiki/Birthday_Paradox), a hash that is long enough to make collisions unlikely would also be long enough to make big gains from hashing unlikely.
There is a considerable body of knowledge on perfect hashing (see http://en.wikipedia.org/wiki/Perfect_hash_function), but it is not clear how to make that work in our application.

In summary, I think we should be careful to include an optimization that might easily double the code size of a CoAP implementation.

Gruesse, Carsten

---

From: Paul Duffy <[email-masked]>
Date: Fri, 11 Jun 2010 22:29:15 -0400

> With a predefined dictionary (such as the one we made in RFC 3485 for RFC 3320), it *might* be useful if we manage to hit the right strings.

That scenario was what I was pondering, but that seems more appropriate 
for a specific application of COAP (not COAP itself), and even then runs 
afoul of RESTful in that fixed, predicatable  URI structure would be 
needed to take advantage of a dictionary.

> Even with a small dictionary (such as the priority 1 subset from RFC 3485) it also still requires a lot of code.

Agreed, but there are application profiles that may be using DEFLATE for 
certificate compression, and I believe DEFLATE is also part of the EXI 
machinery.

---

From: Carsten Bormann <[email-masked]>
Date: Sat, 12 Jun 2010 14:02:41 +0200

On Jun 12, 2010, at 01:26, Carsten Bormann wrote:

> Huffman

Since a discussion about compression is useless without numbers, I did a couple little experiments.
These experiments, of course, depend on the corpus of URIs that are submitted to compression.
Nobody knows how CoAP url-paths are going to look like in the future, so there is no "good" corpus I could have been using.
Lacking one, I just grabbed a random access.log from a web server and used the referer links.
(These are relatively long URIs, and are dominated by searches, so take the following numbers with a large grain of salt -- real world CoAP numbers will be worse.)

First, I determined the lower bound achievable by Huffman-compressing this corpus with a single static table *optimized for this corpus*.  In other words, any real compression ratio with a static table will be worse than that because of the mismatch between the corpus we use for deriving the static table and whatever turns out to be the real world.  (Also, the padding to full bytes further reduces efficiency for shorter URIs.)

Anyway, the number:

uncompressed: 66600, compressed: 46512, ratio: 0.698378378378378

So the best case we can possibly hope out of using any of the base64 tricks etc. (which are essentially just non-optimal Huffman schemes) is ~ 30 % reduction.

As I didn't consider bit-wise Huffman coding very realistic for constrained devices, I also did a nibble-wise (16-ary) Huffman.  Again the static table was optimized for the corpus I have.

uncompressed: 66600, compressed: 50791, ratio: 0.762627627627628

Not much worse, and much simpler code, as there are only sizes 4, 8, and 12.
Still, I'm not thrilled.

For comparison, I also did the RFC 1951 (Deflate *without a header*) case:

uncompressed: 66600, compressed: 57085, ratio: 0.857132132132132

(I didn't try adding a static dictionary, because I have absolutely no idea what we would put there; if you want to suggest one, I'll redo that number; I can also rerun any corpus you have.)
Even with the long URIs from my unrealistic corpus, the gains are much worse than Huffman, as I expected.

My verdict on the optimization opportunity "URI compression": Let's not go there; absolutely not worth it.

Gruesse, Carsten

---

From: Henning Schulzrinne <[email-masked]>
Date: Sat, 12 Jun 2010 08:36:14 -0400

One possibility is to optimize for the case where the same URL is used again and again, which is likely to be common for many small devices (since they only have a small number of resources and devices in a local 6lowpan network are likely to talk to each other a lot). In that case, you could do the following:

- Use the full URL on first use
- The server returns a hint ("URL /home/lights/kitchen this is my URL #5"), as an 8 or 16-bit integer (say), in the response; popular URLs get the short codes
- On subsequent requests, the client uses the 8 or 16-bit URL
- If the client forgets the binding or doesn't care to remember it, it uses the regular URL. The usual caching mechanisms can be applied if you want to get fancy, but that's an implementation matter.
- If the server forgets the table, it returns an error, and the request gets repeated with the usual URL.

I think the problem here is fundamentally different from, say, SIP header compression since the community of nodes is likely to be much more closed, i.e., with high rates of locality.

That's probably about as efficient as you can make it while not requiring generic dictionaries. Like Carsten, I'm not convinced it's worth it, but at least it gets you more than 20-30%.

This is vaguely similar to the bit.ly-style services that are popular in Facebook and Twitter.

Henning

---

From: "[email-masked]" <[email-masked]>
Date: Sat, 12 Jun 2010 22:22:48 +0300

+1 - I like that Henning. 

In addition, a server could let clients know about its popular URLs a
priori in its /.well-known/resources links as alt= or short=. Instead of
thinking of this as a special integer code, we can do this exactly like a
short URL service. The server just assigns multiple URLs to a resource, one
happens to be /5 in addition to /sensors/temp/indoor/centigrade. Or as
Carsten suggested, it could have a special form like /!5. 

But it wouldn't hurt to do 7-bit ASCII packing on URLs anyways? That is 15%
savings by just bit shifting. I agree that more sophisticated algorithms
probably aren't worth it.

Zach

On Sat, 12 Jun 2010 08:36:14 -0400, Henning Schulzrinne
<[email-masked]> wrote:
> One possibility is to optimize for the case where the same URL is used
> again and again, which is likely to be common for many small devices
(since
> they only have a small number of resources and devices in a local 6lowpan
> network are likely to talk to each other a lot). In that case, you could
do
> the following:
> - The server returns a hint ("URL /home/lights/kitchen this is my URL
#5"),
> as an 8 or 16-bit integer (say), in the response; popular URLs get the
> short codes
> - If the client forgets the binding or doesn't care to remember it, it
uses
> the regular URL. The usual caching mechanisms can be applied if you want
to
> get fancy, but that's an implementation matter.
> I think the problem here is fundamentally different from, say, SIP header
> compression since the community of nodes is likely to be much more
closed,
> i.e., with high rates of locality.
> 
> That's probably about as efficient as you can make it while not requiring
> generic dictionaries. Like Carsten, I'm not convinced it's worth it, but
at
> least it gets you more than 20-30%.
> 
>> Nobody knows how CoAP url-paths are going to look like in the future, so
>> there is no "good" corpus I could have been using.
>> (These are relatively long URIs, and are dominated by searches, so take
>> the following numbers with a large grain of salt -- real world CoAP
>> numbers will be worse.)
>> 
>> First, I determined the lower bound achievable by Huffman-compressing
>> this corpus with a single static table *optimized for this corpus*.  In
>> other words, any real compression ratio with a static table will be
worse
>> than that because of the mismatch between the corpus we use for deriving
>> the static table and whatever turns out to be the real world.  (Also,
the
>> padding to full bytes further reduces efficiency for shorter URIs.)
>> 
>> Anyway, the number:
>> 
>> So the best case we can possibly hope out of using any of the base64
>> tricks etc. (which are essentially just non-optimal Huffman schemes) is
~
>> 30 % reduction.
>> 
>> As I didn't consider bit-wise Huffman coding very realistic for
>> constrained devices, I also did a nibble-wise (16-ary) Huffman.  Again
>> the static table was optimized for the corpus I have.
>> 
>> (I didn't try adding a static dictionary, because I have absolutely no
>> idea what we would put there; if you want to suggest one, I'll redo that
>> number; I can also rerun any corpus you have.)

---

From: Carsten Bormann <[email-masked]>
Date: Sat, 12 Jun 2010 22:23:08 +0200

On Jun 12, 2010, at 21:22, [email-masked] wrote:

> That is 15% savings by just bit shifting.

It's 12.5 % best case (if you have an exact multiple of 8 characters).
For /.well-known/resources, the saving is two bytes (i.e., 19 instead of 21 as we are eliding the initial slash), or 9.5 %.
(The 16-ary Huffman code I've cooked up happens to save 6 bytes or 28.5 % in this case.  Much better, but still barely worth it.)

Agreeing on /!r or some such would reduce the URI to a grand total of two bytes, or save 19 bytes = 90.5 % :-)
Heck, just calling it /.well-known/rsrcs has twice the benefit of 7-bit crunching...

Just to cite (and maybe learn from) some precedence: When SIP was defined, some of its header fields were assigned alternative, single-character "compact form" names.
Most SIP implementations send long form names, not compact forms; this has led to a number of interoperability problems when some implementations did use the compact form and then other implementations didn't include full(ly debugged) support for the compact forms.  (IOW, aliasing = having the same feature under two names can cause problems.)
I haven't seen compact forms in the wild (but then I don't spend much time debugging SIP flows), but I'm sure Henning can fill us in on how much these are liked these days.

Gruesse, Carsten

---

From: Henning Schulzrinne <[email-masked]>
Date: Sat, 12 Jun 2010 17:08:45 -0400

> I haven't seen compact forms in the wild (but then I don't spend much time debugging SIP flows), but I'm sure Henning can fill us in on how much these are liked these days.

I'm sure *somebody* uses them, but they have not been popular and have, as Carsten said, led to more interoperability problems, plus the usual issues of equivalency checks (e.g., for signing). I'm pretty sure we would not define them again if given a chance; while they were meant to be an extensible list, no new headers have defined short forms.

I think such tricks also make life more difficult if we want to go for internationalized URI at some point, which I consider more valuable than 15% of hypothetical savings.

Henning

---

From: Brian Frank <[email-masked]>
Date: Sat, 12 Jun 2010 22:02:56 -0400

My vote is to not bother with URI compression.  If you want to use short
compact URIs, then just define your URI space that way.  There is nothing to
say that URIs have to be human friendly.  I suspect that most 6LoWPAN
devices have less than 2^16 addressable resources, in which case you can use
a four byte hex number to create a simple URI space for all your resources.
 Yes it might be nice to shrink that a byte or two, but is it worth the
added complexity?

On Sat, Jun 12, 2010 at 4:23 PM, Carsten Bormann <[email-masked]> wrote:

> For /.well-known/resources, the saving is two bytes (i.e., 19 instead of 21
> as we are eliding the initial slash), or 9.5 %.
> (The 16-ary Huffman code I've cooked up happens to save 6 bytes or 28.5 %
> in this case.  Much better, but still barely worth it.)
>
> Just to cite (and maybe learn from) some precedence: When SIP was defined,
> some of its header fields were assigned alternative, single-character
> "compact form" names.
> Most SIP implementations send long form names, not compact forms; this has
> led to a number of interoperability problems when some implementations did
> use the compact form and then other implementations didn't include full(ly
> debugged) support for the compact forms.  (IOW, aliasing = having the same
> feature under two names can cause problems.)
> I haven't seen compact forms in the wild (but then I don't spend much time
> debugging SIP flows), but I'm sure Henning can fill us in on how much these
> are liked these days.
>
> Gruesse, Carsten
>

---

From: "Angelo P. Castellani" <[email-masked]>
Date: Mon, 14 Jun 2010 10:36:47 +0200

Dear Brian, Carsten, all,

I see your concerns about complexity, which in general I really share.

However the URI compression format #1 is really really simple to
implement on a constrained node.

Without long discussion I quickly implemented the relevant code to
apply URI compression with dictionary [/0-9a-zA-Z], vocabulary is
defined by a small matrix in the encoder/decoder and can be easily
changed. Nodes can easily support multiple dictionaries and use them
depending on the kind of characters on the URI (or something else)
they want to encode.

http://www.dei.unipd.it/~castellani/encode.c
http://www.dei.unipd.it/~castellani/decode.c
(only the marked parts of the code are the actual compression, very few lines)

Compression is up to -25% which is not that bad for a method simple
like this. In my opinion we can define 2-4 dictionaries for URI
compression, the first can be "no compression" leave the URI as it is,
others can be using the dictionary compression format and a predefined
dictionary.

An user can choose how to compress the URI depending on the URI (using
the dictionary that performs better on that URI) or depending if it
implements dictionary compression or not.

Best,
Angelo

On Sun, Jun 13, 2010 at 04:02, Brian Frank <[email-masked]> wrote:
>> For /.well-known/resources, the saving is two bytes (i.e., 19 instead of
>> 21 as we are eliding the initial slash), or 9.5 %.

---

From: Carsten Bormann <[email-masked]>
Date: Mon, 14 Jun 2010 11:42:46 +0200

On Jun 14, 2010, at 10:36, Angelo P. Castellani wrote:

> really simple to
> implement

I couldn't understand why your code should work so I tried it out to ascertain it doesn't.

$ ./encode /.well-known/resources/lightswitchnumber111 | ./decode | od -c
initial length: 43
final length: 35
compressed chars: 41
ratio: 0.813953
0000000    /   .   w   e   l   l   -   k   n   o   w   n   /   r   e   s
0000020    o   u   r   c   e   s   /   l   i   g   h   t   s   w   i   t
0000040    c   h   n   u   m   b   e   r   1   1   1   _  \n            
0000055

Consider that my case in point :-)
This stuff is indeed hard to get right.

(Oh, and using the "nibbles" 16ary Huffman, this compresses to 31.5, i.e., 32 bytes.
And it's much simpler, so I'm sure I could get it right.  I still don't think it's worth it.)

Gruesse, Carsten

---

From: "Angelo P. Castellani" <[email-masked]>
Date: Mon, 14 Jun 2010 12:00:13 +0200

Thanks for the comments and to try out my quickly written programs.

Well actually the encoding worked, there was a small bug in the
decoding stop condition... Check out the updated decode.c version at
the same URL.

I'm sure that a properly cooked static Huffman code can save some more
bytes (not that much), however I believe that Huffman implementation
will be more painful and not "much simpler"; probably I'm wrong can
you argument why you believe it is that simple?

Best,
Angelo

---

From: Arjun Roychowdhury <[email-masked]>
Date: Tue, 15 Jun 2010 09:58:27 -0400

What is the final compression goal here? One of nice things about an HTTP
url is that it is generally understandable and indicates the target resource
correctly. This entire url shortening services add complexity, single point
of failures as well as increase response latency (due to redirects, in the
case the destination server is not the one maintaining the map of short URIs
to full URIs).

regds
arjun

On Sat, Jun 12, 2010 at 3:22 PM, [email-masked] <[email-masked]>wrote:

> *********************************************DISCLAIMER*********************************************
> The email is subject to Disclaimer as per -
> http://www.hsc.com/tabid/91/Default.aspx#emaildisclaimer
>
>
>

-- 
Arjun Roychowdhury

---

From: [email-masked]
Date: Wed, 16 Jun 2010 10:12:27 +0200

Hi all,

>My vote is to not bother with URI compression.  If you want to use short
compact URIs, then just define your URI space that way.  There is nothing
to say that URIs have to be human friendly.  I suspect that most 6LoWPAN
devices have less than 2^16 addressable resources, in which case you can
use a four byte hex number to create a simple URI space for all your
resources.  Yes it might be nice to shrink that a byte or two, but is it
worth the added complexity?

I also agree that a well defined URI space with for example 4 char
names/digits is better than a complex encoding of long paths.
Long names are only useful for humans and CoAP is supposed to be a M2M
procotol.

But what about
coap://[2001:1ba3::ff:fe:0:1]:61617/1234
or
coap://[3ffe:b80:3:1a91:224:e8ff:fe92:5dcf]/1234

The path is short but the authority is verbose.
This aboslute form is still required by a proxy and is quite complex to
parse and to shrink.

So the questions are:
Is it possible to encode the authority part in a dedicated binary option in
CoAP ?
Could we define or reuse prefix contexts in a way ?
What are the supported parts of an URI ? Scheme, user info, host, port ?
What is the minimum length for URIs that a CoAP device must support ?
Currently the only available value in the draft is 32768 if I'm right.

Regards,
Matthieu

             Brian Frank                                                   
             <brian.tridium@gm                                             
             ail.com>                                                    A 
             Envoyé par :              Carsten Bormann <[email-masked]>      
             core-bounces@ietf                                          cc 
             .org                      core issue tracker                  
                                       <[email-masked]>, core         
                                       <[email-masked]>                     
             13/06/2010 04:02                                        Objet 
                                       Re: [core] #4: URI Encoding         

                                                                           

My vote is to not bother with URI compression.  If you want to use short
compact URIs, then just define your URI space that way.  There is nothing
to say that URIs have to be human friendly.  I suspect that most 6LoWPAN
devices have less than 2^16 addressable resources, in which case you can
use a four byte hex number to create a simple URI space for all your
resources.  Yes it might be nice to shrink that a byte or two, but is it
worth the added complexity?

      > That is 15% savings by just bit shifting.

      It's 12.5 % best case (if you have an exact multiple of 8
      characters).
      For /.well-known/resources, the saving is two bytes (i.e., 19 instead
      of 21 as we are eliding the initial slash), or 9.5 %.
      (The 16-ary Huffman code I've cooked up happens to save 6 bytes or
      28.5 % in this case.  Much better, but still barely worth it.)

      Agreeing on /!r or some such would reduce the URI to a grand total of
      two bytes, or save 19 bytes = 90.5 % :-)
      Heck, just calling it /.well-known/rsrcs has twice the benefit of
      7-bit crunching...

      Just to cite (and maybe learn from) some precedence: When SIP was
      defined, some of its header fields were assigned alternative,
      single-character "compact form" names.
      Most SIP implementations send long form names, not compact forms;
      this has led to a number of interoperability problems when some
      implementations did use the compact form and then other
      implementations didn't include full(ly debugged) support for the
      compact forms.  (IOW, aliasing = having the same feature under two
      names can cause problems.)
      I haven't seen compact forms in the wild (but then I don't spend much
      time debugging SIP flows), but I'm sure Henning can fill us in on how
      much these are liked these days.

      Gruesse, Carsten

______________________________________________________________________
This email has been scanned by the MessageLabs Email Security System.
______________________________________________________________________
_______________________________________________
core mailing list
[email-masked]
https://www.ietf.org/mailman/listinfo/core

---

From: "Angelo P. Castellani" <[email-masked]>
Date: Wed, 16 Jun 2010 10:35:41 +0200

Matthieu,

a transparent proxy can make use of the 6LoWPAN compressed CoAP server
IP/port destination, removing the need to define a compressed format
of the request for a proxy.

I actually agree on compressing the resource URI in the CoAP server if
possible (take a look at
http://www.ietf.org/mail-archive/web/core/current/msg00315.html,
compression format #2), defining a short URL has the same complexity
of defining an URI compressed as proposed in #2 which however is more
efficient.

Binary-URIs can also be always represented in 7-bit ASCII with a
simple mapping, so defining a Binary-URI has the side effect of
defining a short URL too.

In my opinion defining a compression for long URIs is useful when we
want to send a request to a legacy HTTP server, which resource names
are not compressed (CoAP constrained client -> HTTP legacy server).

Best,
Angelo

On Wed, Jun 16, 2010 at 10:12,
<[email-masked]> wrote:
>
> Hi all,
>
> >My vote is to not bother with URI compression.  If you want to use short compact URIs, then just define your URI space that way.  There is nothing to say that URIs have to be human friendly.  I suspect that most 6LoWPAN devices have less than 2^16 addressable resources, in which case you can use a four byte hex number to create a simple URI space for all your resources.  Yes it might be nice to shrink that a byte or two, but is it worth the added complexity?
>
> I also agree that a well defined URI space with for example 4 char names/digits is better than a complex encoding of long paths.
> Long names are only useful for humans and CoAP is supposed to be a M2M procotol.
>
> But what about
> This aboslute form is still required by a proxy and is quite complex to parse and to shrink.
>
> So the questions are:
> Is it possible to encode the authority part in a dedicated binary option in CoAP ?
> What is the minimum length for URIs that a CoAP device must support ? Currently the only available value in the draft is 32768 if I'm right.
>
> Regards,
> Matthieu
>
> Brian Frank <[email-masked]>
>
>
>
>
> Brian Frank <[email-masked]>
> Envoyé par : [email-masked]
>
> 13/06/2010 04:02
>
> A
> Carsten Bormann <[email-masked]>
> cc
> core issue tracker <[email-masked]>, core <[email-masked]>
> Objet
> My vote is to not bother with URI compression.  If you want to use short compact URIs, then just define your URI space that way.  There is nothing to say that URIs have to be human friendly.  I suspect that most 6LoWPAN devices have less than 2^16 addressable resources, in which case you can use a four byte hex number to create a simple URI space for all your resources.  Yes it might be nice to shrink that a byte or two, but is it worth the added complexity?
>
>
> _____________________________________________________________________________________________________________________
> core mailing list
> [email-masked]

---

From: Carsten Bormann <[email-masked]>
Date: Wed, 16 Jun 2010 11:19:19 +0200

> Is it possible to encode the authority part in a dedicated binary option in CoAP ?

If we believe access to CoAP URIs via proxies by using CoAP is worth an optimization, we can certainly do that.
However, I'd expect CoAP proxies to gateway to other schemes as well, so I'm not sure how useful that optimization is.

See also issue #12; we need to do some work here anyway.

> Could we define or reuse prefix contexts in a way ?

That is more difficult, as it would presume the proxy has access to a compression context, and that this context is guaranteed to be the same as the one used by the client.  I don't know how to make sure that this is the case.

The right way to do this is to compress at the adaptation layer; some form of compression within the payloads based on prefixes may be useful.  (I have a blueprint for that, but no idea whether it's worth it: We could also simply compress based on the transport addresses used in the packet, which *are* accessible to both nodes.)

> What are the supported parts of an URI ? Scheme, user info, host, port ?

I don't think we do user/password.  Wo obviously do need scheme, host, port.

> What is the minimum length for URIs that a CoAP device must support ? Currently the only available value in the draft is 32768 if I'm right.

The option encoding, as well as the maximum size of a CoAP datagram, limit this further.

I think the last paragraph of section 3.2.1 of RFC 2616 gives a good general idea for how long URIs used to be in the early Web: 

      Note: Servers ought to be cautious about depending on URI lengths
      above 255 bytes, because some older client or proxy
      implementations might not properly support these lengths.

Gruesse, Carsten

---

From: Paul Duffy <[email-masked]>
Date: Wed, 16 Jun 2010 14:07:08 -0400

What exactly do you classify as a "HTTP legacy server"?

On 6/16/2010 4:35 AM, Angelo P. Castellani wrote:
>> Brian Frank<[email-masked]>
>>
>>
>>
>>
>> Brian Frank<[email-masked]>
>> Carsten Bormann<[email-masked]>
>> cc
>> core issue tracker<[email-masked]>, core<[email-masked]>
>> Objet
>> On Sat, Jun 12, 2010 at 4:23 PM, Carsten Bormann<[email-masked]>  wrote:
>>

---

From: "Angelo P. Castellani" <[email-masked]>
Date: Wed, 16 Jun 2010 21:38:38 +0200

A generic HTTP server running on the Internet...

If a node needs to access resources exposed on the Internet by an HTTP
server, it can send the request using CoAP. An edge gateway can
translate the request to HTTP, relay it and when the response gets
back translate it to CoAP and relay it back to the node.

In this use case compressing long URLs can lead to a significant advantage.

In this case however we cannot expect URLs to be always short.

Regards,
Angelo

On Wednesday, June 16, 2010, Paul Duffy <[email-masked]> wrote:
> Most SIP implementations send long form names, not compact forms; this has led to a number of interoperability

---

From: Zach Shelby <[email-masked]>
Date: Thu, 17 Jun 2010 23:02:28 +0300

>      implementations might not properly support these lengths.

I would like it if the limit would be in that range for CoAP. 

Zach

-- 
Zach Shelby, Chief Nerd, Sensinode Ltd.
http://zachshelby.org  - My blog "On the Internet of Things"
http://6lowpan.net - My book "6LoWPAN: The Wireless Embedded Internet"
Mobile: +358 40 7796297

---

From: Zach Shelby <[email-masked]>
Date: Thu, 17 Jun 2010 23:20:52 +0300

> My vote is to not bother with URI compression.  If you want to use short compact URIs, then just define your URI space that way.  There is nothing to say that URIs have to be human friendly.  I sus…

Right, it starts to be clear that we need to support short alternative URIs. In theory that doesn't need anything special. A resource provider simply assigns an alternative (short) URI of its own choosing to resources that have long URIs or are popular. So short URIs in CoAP don't require any kind of mapping server, nor much extra state.

What do we need to make that work even better? How does a requester find out about a short URI? This could be enough:

1. Henning threw out the idea of the provider being able to include a short URI for a resource in the response. So if a provider receives a request for the normal URI of a resource, it may include a short alternative URI in the response message (if any). After that the requestor can use the alternative URI in subsequent requests. This is easy to add as the response simply includes a CoAP Uri option in that case. 

2. We could include an attribute "alt=" in the resource description format, which allows a requestor to discover the alternative URI of a resource a priori. 

What do people think? If the WG likes the idea we can make a ticket for us to flesh out the details.

This is of course orthogonal to the encoding discussion. 

Zach

---

From: Cullen Jennings <[email-masked]>
Date: Mon, 21 Jun 2010 10:45:39 -0600

Hennings suggestion seemed well worth fleshing out and seeing what it looks like - look appealing. 

I'd like to add a variant to it of also allowing  a IANA FCFS registry to assign the numbers to URI. Perhaps a variably length encoded 15 or 23 bit number. It gets to the point that these URI  are for M2M communications and human readability is less important. 

Cullen ... Comments as an individual contributor ...

>> I haven't seen compact forms in the wild (but then I don't spend much time debugging SIP flows), but I'm sure Henning can fill us in on how much these are liked these days.

Cullen Jennings
For corporate legal information go to:
http://www.cisco.com/web/about/doing_business/legal/cri/index.html