Subject: [Simple] RE: XCAP Directory: Large response ?

From: "Henrik Albertsson (KI/EAB)" <[email-masked]>
Date: Fri, 2 Jul 2004 10:08:54 +0200

Isn't this also a problem for XCAP in general - you do not know the size of the list you are retrieving. You might select the wrong list and fetch the global contact list containing all Nokia employees, then you would at least need a way to get the list in manageable pieces. I see the options from a requirement point as: 

(1) You know the size of the document you are retrieving and then decide to do it or not. Most basic is the data size of the list or more advanced the number of elements in the list. 

(2) You can indicate how many items of the list you want to retrieve, you get for example the first 50 - then nothing more. 

(3) You can indicate how many items you want in each "slice", then you can iterative go and fetch the list in the sizes you want. 

Any options I have missed? 

/Henrik 

-----Original Message-----
From: Miguel Garcia [mailto:[email-masked]]
Sent: den 2 juli 2004 09:15
To: Henrik Albertsson (KI/EAB)
Cc: [email-masked]
Subject: Re: XCap Directory: Large respons ?

Hi Henrik:

I think you got a good point. I think we need to add a new requirement 
to the draft that reads something like:

- The XCAP client MUST be able to limit the amount of information 
returned by the XCAP server, so that the XCAP directory document does 
not flood the client.

As for a possible solution... I need to think about it... Most likely we 
will need to define some sort of convention, kind of adding a parameter 
to the URI to limit the number of entries in the response. Something like:

GET http://xcap.example.com/services/directory/users/
bill/directory.xml?entries=50 HTTP/1.1

I am open to hear suggestions.

Regards,

          Miguel

Henrik Albertsson (KI/EAB) wrote:
> Hi,
> 
> The current version of the XCAP Directory does not contain any limit of the size of the response. An XCAP Directory can potentially be very large if it collects all list for a user, especially in a mobile environment this would be problem. Any ideas how to limit the response? 
> 
> Regards
> /Henrik 
> 
>  
>  
> Henrik Albertsson 
> System Manager
> IMS Service Layer
> 
>  
> 
> Multimedia Solutions System Management
> Ericsson AB
> S-125 26 Stockholm 
> Sweden
> Tel: +46 8 719 90 73
> Mobile: +46 705 19 85 39
> Fax: +46  8 404 92 22
> Visiting address: Kistagången 26
> 

-- 
Miguel A. Garcia           tel:+358-50-4804586
Nokia Research Center      Helsinki, Finland

_______________________________________________
Simple mailing list
[email-masked]
https://www1.ietf.org/mailman/listinfo/simple

---

From: Miguel Garcia <[email-masked]>
Date: Fri, 02 Jul 2004 11:50:42 +0300

Henrik (and Jonathan):

I think you are right, the problem of the large lists is not only 
applicable to the XCAP directory draft, but to the basic XCAP.

Jonathan, do the basic XCAP or the resource list application usage have 
a mechanism to limit the number of entries returned in the list? I think 
this is a must in wireless environments.

Henrik proposed three solutions, for which 2 and 3 are the same. I 
believe we need to add some "range" parameter to XCAP fetch operations 
to limit the result. So, if I send a GET xcapDocument.xml?range=1-50, 
the XCAP server will return up to 50 entries in the list. Then I can do 
a new GET with the next range, e.g., GET xcapDocument.xml?range=51-100, 
and so on.

How about that?

- Miguel

---

From: Jonathan Rosenberg <[email-masked]>
Date: Fri, 02 Jul 2004 06:00:49 -0400

inline.

> this is a must in wireless environments.

HTTP already provides a feature to do this, and since XCAP is just a 
usage of HTTP, that feature is inherited. The Range header field allows 
a client to request, amongst other things, a specific maximum number of 
bytes that it wants of the entity body. Quoting from RFC2616:

> By its choice of last-byte-pos, a client can limit the number of
>    bytes retrieved without knowing the size of the entity.

Byte ranges also allow the client to fetch a large resource in pieces, 
or to give up after obtaining the first piece.

HTTP also provides the HEAD method, which returns all of the meta-data 
about an entity, including its length (I believe), but not the entity 
itself. Thus, if you were worried about this, you could do a HEAD for 
the resource, see how bit it is, and then GET it if you want it.

> a new GET with the next range, e.g., GET xcapDocument.xml?range=51-100, 

I would propose that we not put this functionality in xcap itself, but 
use the existing HTTP mechanisms. I will note that these mechanisms are 
*byte* oriented, and not XML oriented. Thus, you may cut an element off 
in the middle by asking for the first 50 bytes. However, if your main 
concern is bandwidth or latency, then bytes are much more significant of 
a measure than elements. If I say, "give me 30 elements", that could be 
30 bytes or 30MB, depennding on the amount of data in each element.

-Jonathan R.

-- 
Jonathan D. Rosenberg, Ph.D.                600 Lanidex Plaza
Chief Technology Officer                    Parsippany, NJ 07054-2711
dynamicsoft
[email-masked]                     FAX:   (973) 952-5050
http://www.jdrosen.net                      PHONE: (973) 952-5000
http://www.dynamicsoft.com

---

From: Miguel Garcia <[email-masked]>
Date: Fri, 02 Jul 2004 13:14:17 +0300

> 30 bytes or 30MB, depennding on the amount of data in each element.

Jonathan, I think the HTTP byte ranges will not work. Let's say I 
request a long document, and I request bytes 1 to 10000. The HTTP server 
will cut the XML document exacltly when the number of bytes has reached 
the count, and will send it to the XCAP client. The XCAP client can't 
check if the document is well-formed, not even check if it is valid 
according to the XML schema. As a consequence, it won't display it to 
the user, or dare to do any operation on it. The only thing the XCAP 
client can do is to request the next byte range.

I am concerned with the limited amount of memory a device might have, 
AND the bandwidth. The latter can be solved with byte ranges, but not 
the former.

However, if we provide a mechanism to request ranges of entries in a 
list, this will solve the problem: the XCAP server create a valid XML 
document that complies with the schema and send it to the XCAP client. 
The client can request additional ranges once it has freed the memory. 
The drawback is that we cannot accurately control the bandwidth (we 
don't know how many bytes will come), but I don't think this is a big issue.

- Miguel

---

From: Ted Hardie <[email-masked]>
Date: Fri, 2 Jul 2004 11:21:11 -0700

At 6:00 AM -0400 7/2/04, Jonathan Rosenberg wrote:
>
>I would propose that we not put this functionality in xcap itself, 
>but use the existing HTTP mechanisms. I will note that these 
>mechanisms are *byte* oriented, and not XML oriented. Thus, you may 
>cut an element off in the middle by asking for the first 50 bytes. 
>However, if your main concern is bandwidth or latency, then bytes 
>are much more significant of a measure than elements. If I say, 
>"give me 30 elements", that could be 30 bytes or 30MB, depennding on 
>the amount of data in each element.

I'd like to strongly support this view.  XCAP is currently a profile of
HTTP, and re-using HTTP methods makes a great deal of sense here.
Attempting to extend XCAP to include mechanisms that are not part
of the core of HTTP will require either extending HTTP (hard)
or diverging from HTTP (problematic for code re-use, which has
been a key factor before).

I'd also like to point out that the problem (unexpectedly large responses)
may be less likely than originally supposed.  An XCAP client will have
to have some knowledge of the schema with which it is interacting
in order to get the base advantage of XCAP over HTTP.  That is, it
will have to know the break points between things like the whole
buddy list, the list of "friends", the subset of "friends" who are
"good friends", and a possibly overlapping set of "work colleagues".
While it would not be formally part of the schema, it would not be
hard to store the number of elements last known for each of the
break points in the hierarchy; indeed, the previous discussion of
positional insertions seemed to presume this was stored.   Even
without positional insertions, this is useful information that it is
relatively easy to store and maintain.  As Jonathan points out,
it still won't map perfection to bytes, since elements may be of
different sizes, but it will help the client understand the likely
size of a request.

		regards,
				Ted Hardie

---

From: Miguel Garcia <[email-masked]>
Date: Mon, 05 Jul 2004 10:42:39 +0300

I think the discussion is becoming clearer now. For sure an XCAP client 
will have knowledge of the schema is trying to download/manipulate, so 
in most circumstances it can just request to download a few elements 
(e.g., the "good friends" list).

The problems is when I get my brand new mobile device, and I want to add 
a new buddy to my list. First I need to download my complete presence 
list. If that list is huge (well, imagine my companies employees list), 
then I may flood my terminal (or even the access network).

The Range header in HTTP does not seem to be quite useful, since it will 
truncate the XML document. Then the XCAP client cannot validate it, it 
can either request the rest of the document or abort the operation.

The position() XPath operator seems to fulfill the requirements. I can 
request entries 1 to 50 to avoid flooding my terminal. Since the XCAP 
server will return a valid XML document, it can be displayed to the 
user, and do some manipulation. Then the XCAP client can request entries 
51 to 100, etc.

Yes, we don't know how many bytes will be sent to the terminal in each 
of the pages, but I don't think the exact number of bytes is so 
important, what is important is the number of entries in a list.

- Miguel

Ted Hardie wrote:

>> use the existing HTTP mechanisms. I will note that these mechanisms 
>> are *byte* oriented, and not XML oriented. Thus, you may cut an 
>> element off in the middle by asking for the first 50 bytes. However, 
>> if your main concern is bandwidth or latency, then bytes are much more 
>> significant of a measure than elements. If I say, "give me 30 
>> elements", that could be 30 bytes or 30MB, depennding on the amount of 
>> data in each element.
> 
>

---

From: Ted Hardie <[email-masked]>
Date: Mon, 5 Jul 2004 11:36:55 -0700

At 10:42 AM +0300 7/5/04, Miguel Garcia wrote:
>
>The Range header in HTTP does not seem to be quite useful, since it 
>will truncate the XML document. Then the XCAP client cannot validate 
>it, it can either request the rest of the document or abort the 
>operation.

Under normal circumstances, I would expect the range header to be used
to collect the whole of the document, which the terminal would then process.
If it cannot store the whole document or process it, it seems to me it must
understand the schema in order to proceed (using XPATH or the hierarchy
inherent in XCAP).

			regards,
				Ted Hardie