Subject: [Simple] RE: XCAP Directory: Large response ?

From: "Henrik Albertsson (KI/EAB)" <[email-masked]>
Date: Fri, 2 Jul 2004 12:37:31 +0200

I agree with you Miguel that the "give me X byte" solution is not usable for the reasons you mentioned. However most likely the XCAP client will operate on a well known AUDI and have knowledge about what the structure of the XCAP document is so it can roughly estimate what size 30 elements would be. 

Is it so that when specifying a XCAP document we also need to define what element will be used for indexing when asking for a range of elements? 

Regards
/Henrik 

> -----Original Message-----
> From: Miguel Garcia [mailto:[email-masked]]
> Sent: den 2 juli 2004 12:14
> To: ext Jonathan Rosenberg
> Cc: Henrik Albertsson (KI/EAB); [email-masked]
> Subject: Re: XCAP Directory: Large response ?
> 
> 
> Jonathan Rosenberg wrote:
> 
> > 
> > I would propose that we not put this functionality in xcap 
> itself, but 
> > use the existing HTTP mechanisms. I will note that these 
> mechanisms are 
> > *byte* oriented, and not XML oriented. Thus, you may cut an 
> element off 
> > in the middle by asking for the first 50 bytes. However, if 
> your main 
> > concern is bandwidth or latency, then bytes are much more 
> significant of 
> > a measure than elements. If I say, "give me 30 elements", 
> that could be 
> > 30 bytes or 30MB, depennding on the amount of data in each element.
> > 
> > -Jonathan R.
> > 
> 
> Jonathan, I think the HTTP byte ranges will not work. Let's say I 
> request a long document, and I request bytes 1 to 10000. The 
> HTTP server 
> will cut the XML document exacltly when the number of bytes 
> has reached 
> the count, and will send it to the XCAP client. The XCAP client can't 
> check if the document is well-formed, not even check if it is valid 
> according to the XML schema. As a consequence, it won't display it to 
> the user, or dare to do any operation on it. The only thing the XCAP 
> client can do is to request the next byte range.
> 
> I am concerned with the limited amount of memory a device might have, 
> AND the bandwidth. The latter can be solved with byte ranges, but not 
> the former.
> 
> However, if we provide a mechanism to request ranges of entries in a 
> list, this will solve the problem: the XCAP server create a valid XML 
> document that complies with the schema and send it to the 
> XCAP client. 
> The client can request additional ranges once it has freed 
> the memory. 
> The drawback is that we cannot accurately control the bandwidth (we 
> don't know how many bytes will come), but I don't think this 
> is a big issue.
> 
> - Miguel
> 
> -- 
> Miguel A. Garcia           tel:+358-50-4804586
> Nokia Research Center      Helsinki, Finland
> 

_______________________________________________
Simple mailing list
[email-masked]
https://www1.ietf.org/mailman/listinfo/simple

---

From: [email-masked]
Date: Mon, 5 Jul 2004 14:48:14 +0300

> From: [email-masked] 
> [mailto:[email-masked]]On Behalf
> Of ext Miguel Garcia
> Sent: 05.July.2004 12:50
> To: Khartabil Hisham (Nokia-TP-MSW/Helsinki)
> Cc: Ted Hardie; Jonathan Rosenberg; [email-masked]
> Subject: Re: [Simple] Re: XCAP Directory: Large response ?
> 
> 
> I am writing a set of comments to the filter format, but when 
> I read the 
> document I thought it was only applicable to SIP events. At least the 
> Abstract and the Introduction consider only the case where the filter 
> applies to SIP Notifications.
> 
> However, in this thread, we were discussing XCAP documents that you 
> retrieve with an HTTP GET.
> 
> Are you suggesting that Filter Criteria documents can be applied to 
> XCAP? Is this discussed somewhere?

It's not discussed in the draft, but I was suggesting it can be applied (the XML format)

/Hisham

> 
> - Miguel
> 
> Khartabil Hisham (Nokia-TP-MSW/Helsinki) wrote:
> > I wonder if you can use the following event filtering draft 
> with HTTP GET?
> > 
> > 
> http://www.ietf.org/internet-drafts/draft-ietf-simple-filter-f
> ormat-01.txt (currently in WGLC)
> > 
> > At least the <what> part could be useful.
> > 
> > /Hisham
> > 
> > 
> >>From: [email-masked] 
> >>[mailto:[email-masked]]On Behalf
> >>Of ext Miguel Garcia
> >>Sent: 05.July.2004 10:43
> >>To: Ted Hardie
> >>Cc: Jonathan Rosenberg; [email-masked]
> >>Subject: Re: [Simple] Re: XCAP Directory: Large response ?
> >>
> >>
> >>I think the discussion is becoming clearer now. For sure an 
> >>XCAP client 
> >>will have knowledge of the schema is trying to 
> >>download/manipulate, so 
> >>in most circumstances it can just request to download a few 
> elements 
> >>(e.g., the "good friends" list).
> >>
> >>The problems is when I get my brand new mobile device, and I 
> >>want to add 
> >>a new buddy to my list. First I need to download my 
> complete presence 
> >>list. If that list is huge (well, imagine my companies 
> >>employees list), 
> >>then I may flood my terminal (or even the access network).
> >>
> >>The Range header in HTTP does not seem to be quite useful, 
> >>since it will 
> >>truncate the XML document. Then the XCAP client cannot 
> >>validate it, it 
> >>can either request the rest of the document or abort the operation.
> >>
> >>The position() XPath operator seems to fulfill the 
> >>requirements. I can 
> >>request entries 1 to 50 to avoid flooding my terminal. 
> Since the XCAP 
> >>server will return a valid XML document, it can be displayed to the 
> >>user, and do some manipulation. Then the XCAP client can 
> >>request entries 
> >>51 to 100, etc.
> >>
> >>Yes, we don't know how many bytes will be sent to the 
> >>terminal in each 
> >>of the pages, but I don't think the exact number of bytes is so 
> >>important, what is important is the number of entries in a list.
> >>
> >>- Miguel
> >>
> >>Ted Hardie wrote:
> >>
> >>
> >>>At 6:00 AM -0400 7/2/04, Jonathan Rosenberg wrote:
> >>>
> >>>
> >>mechanisms 
> >>>>are *byte* oriented, and not XML oriented. Thus, you may cut an 
> >>>>element off in the middle by asking for the first 50 
> >>
> >>bytes. However, 
> >>
> >>>>if your main concern is bandwidth or latency, then bytes 
> >>
> >>are much more 
> >>
> >>>>significant of a measure than elements. If I say, "give me 30 
> >>>>elements", that could be 30 bytes or 30MB, depennding on 
> >>
> >>the amount of 
> >>
> >>>>data in each element.
> >>>
> >>>
> >>>I'd like to strongly support this view.  XCAP is currently 
> >>
> >>a profile of
> >>
> >>>HTTP, and re-using HTTP methods makes a great deal of sense here.
> >>>Attempting to extend XCAP to include mechanisms that are not part
> >>>of the core of HTTP will require either extending HTTP (hard)
> >>>or diverging from HTTP (problematic for code re-use, which has
> >>>been a key factor before).
> >>>
> >>>I'd also like to point out that the problem (unexpectedly 
> >>
> >>large responses)
> >>
> >>>may be less likely than originally supposed.  An XCAP 
> >>
> >>client will have
> >>
> >>>to have some knowledge of the schema with which it is interacting
> >>>in order to get the base advantage of XCAP over HTTP.  That is, it
> >>>will have to know the break points between things like the whole
> >>>buddy list, the list of "friends", the subset of "friends" who are
> >>>"good friends", and a possibly overlapping set of "work 
> colleagues".
> >>>While it would not be formally part of the schema, it would not be
> >>>hard to store the number of elements last known for each of the
> >>>break points in the hierarchy; indeed, the previous discussion of
> >>>positional insertions seemed to presume this was stored.   Even
> >>>without positional insertions, this is useful information 
> that it is
> >>>relatively easy to store and maintain.  As Jonathan points out,
> >>>it still won't map perfection to bytes, since elements may be of
> >>>different sizes, but it will help the client understand the likely
> >>>size of a request.
> >>>
> >>>        regards,
> >>>                Ted Hardie
> >>
> >>--

---

From: "George Foti (QA/EMC)" <[email-masked]>
Date: Tue, 6 Jul 2004 06:33:36 -0500

Hisham, you are assuming that the user remembers the list names,  etc..
How realistic is that ??

Rgds/gf
> From: [email-masked] [mailto:[email-masked]]
> Sent: Tuesday, July 06, 2004 7:39 AM
> To: [email-masked]
> Cc: [email-masked]; George Foti (QA/EMC); [email-masked]
> Subject: RE: [Simple] RE: XCAP Directory: Large response ?
> 
> 
> I think extending XCAP to enable this feature is limiting and 
> only works for XCAP. A better solution would be to create a 
> general HTTP solution.
> 
> I acknowledge that the filter stuff as it is defined now does 
> not allow you to do what you want, but we can extend it in a 
> similar way Jonathan proposed to extend XCAP. I.e. Allow the 
> filter to carry the position() function. The example I gave 
> therefore becomes:
> 
>    GET ...
> 
>    <?xml version="1.0" encoding="UTF-8"?>
>    <filter-set xmlns="urn:ietf:params:xml:ns:simple-filter">
>      <filter id="8439" uri="sip:[email-masked]">
>        <what>
>          <include>
>            
> //resource-lists/list[@name="friends"]/entry[position() > N 
> and position() < M]
>          </include>
>        </what>
>        </filter>
>    </filter-set>
> 
> Regards,
> Hisham
> 
> > Sent: 06.July.2004 13:51
> > Cc: Jonathan Rosenberg; George Foti (QA/EMC); [email-masked]
> > Subject: Re: [Simple] RE: XCAP Directory: Large response ?
> > 
> > 
> > Well, the benefit is clear to me. Let's assume you can 
> > accommodate 100 
> > resources in a device. You get resources 1-100, do some xcap 
> > operation 
> > (add, delete, modify), then you can release your memory and get 
> > resources 101-200, and so on.
> > 
> > Think for a second that this is something similar to browsing your 
> > e-mails through a web interface. All the applications I am aware of 
> > allow you to do some pagination. They return a valid HTML 
> code with a 
> > range of messages of your inbox. The key feature is to avoid 
> > flood your 
> > browser and get the user bored scrolling down over 1000 messages.
> > 
> > - Miguel
> > 
> > > I really wonder what the benefit is from such feature. If 
> > there are 1000 resources in a resource list and I can only 
> > fit 100 due to memory limitations in my device. What would it 
> > mean for me to get the first 100, the 2nd 100 or the 
> > resources between 500 and 599?
> > > 
> > > I would rather prefer a feature where I can select the 
> > resources I want to get, if they are indeed on the list. 
> > That's why I was suggestion an event-filter like approach we 
> > I select the resources explicitly.
> > > 
> > > I gave an example on this in a separate thread, but I copy 
> > it again here:
> > > 
> > >    GET ...
> > > 
> > >    <?xml version="1.0" encoding="UTF-8"?>
> > >    <filter-set xmlns="urn:ietf:params:xml:ns:simple-filter">
> > >      <filter id="8439" uri="sip:[email-masked]">
> > >        <what>
> > >          <include>
> > >            
> > 
> //resource-lists/list[@name="friends"]/entry[@uri=""[email-masked]"]
> > >          </include>
> > >          <include>
> > >            
> > //resource-lists/list[@name="friends"]/entry[@uri=""alice@exam
> > ple.com"]
> > >          </include>
> > >        </what>
> > >        </filter>
> > >    </filter-set>
> > > 
> > > 
> > > So a filter is used to GET the resources a client is 
> > interested in, if they are present on the list.
> > > 
> > > /Hisham
> > > 
> > > 
> > >>Sent: 06.July.2004 10:16
> > >>To: Jonathan Rosenberg
> > >>Cc: George Foti (QA/EMC); [email-masked]
> > >>Subject: Re: [Simple] RE: XCAP Directory: Large response ?
> > >>
> > >>
> > >>Summary of the thread:
> > >>
> > >>- I think we have consensus on adding a feature that 
> allows an XCAP 
> > >>client to request a range of XML elements in a particular schema.
> > >>
> > >>- It seems that the HTP Range header does not serve for the 
> > purpose, 
> > >>since it operates on bytes and will most likely truncate XML 
> > >>documents.
> > >>
> > >>- It seems that the XPath position() operator will almost 
> > >>serve, but it 
> > >>is not part of the XCAP base specification.
> > >>
> > >>- And we don't want to delay the XCAP base spec by adding 
> > >>more features 
> > >>at this stage.
> > >>
> > >>- So this feature can be added as an advanced XCAP feature at 
> > >>a later stage.
> > >>
> > >>So far so good. At least I agree with the above. There is one 
> > >>bit where 
> > >>I don't quite agree: it seems that the position() opeator 
> in XPath 
> > >>operates on the element level, meaning that if we have a 
> > >>non-flat list, 
> > >>the XCAP client will need to request ranges in each of the 
> > lists (see 
> > >>point 2 below).
> > >>
> > >>I think this is bad, and since we are going to design 
> this extended 
> > >>feature, we should make it simple to use.
> > >>
> > >>First, if I have a device that can accommodate x entries, my 
> > >>XCAP client 
> > >>would need to request x entries for the first list, get the 
> > response 
> > >>(say y entries came), then do a new query requesting x-y 
> > >>entries in the 
> > >>next list, etc. Alternatively, my XCAP client can request x 
> > >>entries en 
> > >>each list, and perhaps I will get n * x entries (n is the 
> > >>number of lists).
> > >>
> > >>Second, this way of working assumes that I know in advance 
> > >>the number of 
> > >>lists that are nested. This assumption is false when I have never 
> > >>downloaded the XML document (such as when I start using a 
> > new device).
> > >>
> > >>In short: I want to have the means in XCAP to indicate: 
> > >>"please send me 
> > >>this XML document with range of x to z entries, and send them 
> > >>to me in a 
> > >>valid XML document". I don't need to know the structure of 
> > >>the document 
> > >>in advance (how many deep nested lists and their names), 
> > >>although I know 
> > >>the XML schema of the document.
> > >>
> > >>Is this something we can agree upon?
> > >>
> > >>- Miguel
> > >>
> > >>>inline.
> > >>>Miguel Garcia wrote:
> > >>>>>XCAP doesnt support that level of complexity in the 
> > server. If it 
> > >>>>>did, and you wanted protection against memory overload 
> > >>
> > >>(say, 1k max), 
> > >>
> > >>>>>you'd do something like this:
> > >>>>>
> > >>>>>GET http://server/blah-blah/list/buddy[position() > N and 
> > >>
> > >>position() 
> > >>
> > >>>>>< M]
> > >>>>>Range: bytes=0-1000
> > >>>>>
> > >>>>>If the response is truncated, then you need to adjust 
> > >>
> > >>your pagination 
> > >>
> > >>>>>size on the client.
> > >>>>>
> > >>>>>-Jonathan R.
> > >>>>>
> > >>>>
> > >>>>I think this is going in the right direction now. So 
> let me ask a 
> > >>>>couple of questions to verify that the position() is 
> > valid for the 
> > >>>>purpose we want.
> > >>>>
> > >>>>1) Is position already part of XCAP? By reading the XCAP 
> > >>
> > >>base document 
> > >>
> > >>>>I guess the answer is yes.
> > >>>
> > >>>
> > >>>No, it is not. I would not suggest putting this in the base 
> > >>
> > >>spec, which 
> > >>
> > >>>is already overdue and suffering feature creep. I have 
> > >>
> > >>suggested, in 
> > >>
> > >>>another thread, that we perhaps can have an "xcap advanced 
> > >>
> > >>operations" 
> > >>
> > >>>spec that extended xcap with new features, including multiple 
> > >>>insertions. This would possibly be a candidate feature 
> for such an 
> > >>>extension, if the group decided to pursue that.
> > >>>
> > >>>
> > >>>
> > >>>>2) I guess if we have a non-flat list, the position 
> element will 
> > >>>>operate  in all the leaves of the XML document. For 
> > >>
> > >>instance, if we 
> > >>
> > >>>>have a resource list document, with several <list> 
> > >>
> > >>elements, each one 
> > >>
> > >>>>containing a few <entry> elements, the position() 
> > >>
> > >>predicate executed 
> > >>
> > >>>>in the <entry> element will consider the <entry> element 
> > >>
> > >>sequentially 
> > >>
> > >>>>ordered.
> > >>>
> > >>>
> > >>>No. It would apply only at the specific place in the 
> > >>
> > >>hierarchy where it 
> > >>
> > >>>is placed.
> > >>>
> > >>>
> > >>>>Let me try to explain better with an example. Consider the 
> > >>
> > >>following 
> > >>
> > >>>>resource list XML document.
> > >>>>
> > >>>>  <list name="friends" uri="sip:[email-masked]" 
> > >>>>subscribeable="true">
> > >>>>    <entry name="Bill" uri="sip:[email-masked]">
> > >>>>      <display-name>Bill Doe</display-name>
> > >>>>    </entry>
> > >>>>    <entry name="Johanna" uri="sip:[email-masked]">
> > >>>>      <display-name>Johanna Doe</display-name>
> > >>>>    </entry>
> > >>>>    <list name="close-friends" 
> uri="sip:[email-masked]"
> > >>>>          subscribeable="true">
> > >>>>      <entry name="Joe" uri="sip:[email-masked]">
> > >>>>        <display-name>Joe Smith</display-name>
> > >>>>      </entry>
> > >>>>      <entry name="Nancy" uri="sip:[email-masked]">
> > >>>>        <display-name>Nancy Gross</display-name>
> > >>>>      </entry>
> > >>>>      <entry name="Peter" uri="sip:[email-masked]">
> > >>>>        <display-name>Peter Black</display-name>
> > >>>>      </entry>
> > >>>>      
> > >>
> > >><external>http://www.example.org/xcap/resource-lists/users/a/foo
> > >>
> > >>>>         </external>
> > >>>>    </list>
> > >>>>  </list>
> > >>>>
> > >>>>There are 5 <entry> elements, named Bill, Johanna, Joe, 
> Nancy and 
> > >>>>Peter. Imagine I want entries 2 to 4 (in spite they span across 
> > >>>>different lists). Will the HTTP URI be something like:
> > >>>>
> > >>>>GET 
> > >>>>
> > > 
> > > 
> > 
> http://server/blah-blah/resource-lists/users/jack/entry[position()>1 
> > > 
> > >>>and position()<5]
> > >>
> > >>
> > >>No, this will not work. I would suggest that, if you are 
> > worried about 
> > >>this, the client would page in each level of the hierarchy 
> > at a time.
> > >>
> > >>
> > >>
> > >>>3) I believe the HTTP byte ranges is a complement of this 
> > feature, a 
> > >>>kind of emergency life jacket that under normal 
> > circumstances will not 
> > >>>be use. I doubt an XCAP client can do much with an XML 
> > document that 
> > >>>is truncated by the byte ranges: it can just discard the 
> > contents or 
> > >>>store them (if it has enough memory) and request the rest of the 
> > >>>document.
> > >>
> > >>
> > >>Right. The problem Ranges solves is if a fixed memory is 
> > available in 
> > >>the device for the content, you can make sure that there 
> is room to 
> > >>store the results. If the results are a truncated document, 
> > you would 
> > >>need to retry with a much smaller page size.
> > >>
> > >>-Jonathan R.
> > >>
> > >>
> > > 
> > > 
> > 
> > --

---

From: [email-masked]
Date: Wed, 7 Jul 2004 11:20:03 +0300

> Sent: 07.July.2004 09:41
> To: Joel M. Halpern
> Cc: [email-masked]
> > Now, if we add the filter definition Hisham has suggested, 
> we at least 
> > have a behaviors that is well defined and can produce meaningful 
> > results.  It even matches a useful and understandable 
> problem.  However, 
> > it is a big change in the semantics of HTTP GET as far as I 
> can tell.  
> > It starts falling into the trap of using HTTP as a generic 
> transport.  
> > Will it work through relays?
> 
> It will work with proxies, although I agree that is extending somehow 
> the semantics of the GET method in HTTP. Having said that, I 
> don't see 
> Hisham's proposal fulfills the requirements. It assumes that 
> you know in 
> advance the XML document, and then you apply a filter. It 
> requires you 
> to know the values you are trying to filter. We don't want to filter 
> based on values, but based on the number of <entry> elements 
> present in 
> the XML document. Therefore, I don't think adding a filter to the GET 
> request will solve the problem.

I did propose this in an earlier email, it is very similar to the URI in the GET approach suggested Jonathan. Here is the text again:

we can extend it in a similar way Jonathan proposed to extend XCAP. I.e. Allow the filter to carry the position() function. The example I gave therefore becomes:

   GET ...

   <?xml version="1.0" encoding="UTF-8"?>
   <filter-set xmlns="urn:ietf:params:xml:ns:simple-filter">
     <filter id="8439" uri="sip:[email-masked]">
       <what>
         <include>
           //resource-lists/list[@name="friends"]/entry[position() > N and position() < M]
         </include>
       </what>
       </filter>
   </filter-set>

It might be overloading the semantics of HTTP GET and it might be too much overhead, but it's a proposal in any case.

Regards,
Hisham