Subject: RE: [Simple] RE: XCAP Directory: Large response ?

From: "George Foti (QA/EMC)" <[email-masked]>
Date: Fri, 2 Jul 2004 09:53:03 -0500

Comments in line Miguel.

> -----Original Message-----
> From: Miguel Garcia [mailto:[email-masked]]
> Sent: Friday, July 02, 2004 9:03 AM
> To: George Foti (QA/EMC)
> Cc: Henrik Albertsson (KI/EAB); Jonathan Rosenberg; [email-masked]
> Subject: Re: [Simple] RE: XCAP Directory: Large response ?
> 
> 
> George:
> 
> I don't like this solution. It requires the XCAP server to do 
> a "trial 
> and error" of the XML document that will be send to the user. 
> Build an 
> XML document with "n" entries, if the size of the document exceeds X 
> bytes, start reducing the number entries, one by one, until 
> the result 
> is less than X bytes. That's awful.
>
It is a managed complexity, and mostly implementation.
With some meta data, it can be manageable.

> I believe we have two constraints here: one is the display 
> size of the 
> device and the other is the memory of the device. I believe a 
> device may 
> want to get a page of entries it can render in the display, 
> then release 
> the memory, and request a new page.
> 
I addition to what you state, anything that is returned has to be usable and we can do operations n them.

> I don't think this can be achieved by selecting the size in 
> bytes, but 
> with the number of entries in a list.
> 
That is true in certain cases not all cases.
We cannot put a solution oly for those cases.
The most common case is where the user dont know what to expect, i.e he or she has no clue on how many entries are in there.

I am afraid that there are no simple solutions unless we do away with some requirements.

> - Miguel
> 
> 
> George Foti (QA/EMC) wrote:
> 
> > A potential solution is for a client to be able to ask 
> either of the following questions:
> > 1) Give me a *maximum* of X bytes ranges Z onwards
> > 2) Give me a *maximum* of X bytes ranges Y-Z
> > 
> > The XCAP server should return in a successfull response an 
> XML document that is schema compliant that does not exceed 
> the *maximum* range. The client can then perform operations 
> on such a document.
> > A request could fail for multiple reasons, no document 
> meets the maximum length, server does not understand the 
> schema, etcc..
> > 
> > The above implies that XCAP servers must understand 
> different schemas, but that may be unavoidable for some operations.
> > The other thing, I dont know if HTTP headers do allow for the above.
> > 
> > Rgds/gf
> > 
> >  
> > 
> > 
> >>-----Original Message-----
> >>From: [email-masked] 
> >>[mailto:[email-masked]]On Behalf
> >>Of Henrik Albertsson (KI/EAB)
> >>Sent: Friday, July 02, 2004 6:38 AM
> >>To: 'Miguel Garcia'; ext Jonathan Rosenberg
> >>Cc: [email-masked]
> >>Subject: [Simple] RE: XCAP Directory: Large response ?
> >>
> >>
> >>I agree with you Miguel that the "give me X byte" solution is 
> >>not usable for the reasons you mentioned. However most likely 
> >>the XCAP client will operate on a well known AUDI and have 
> >>knowledge about what the structure of the XCAP document is so 
> >>it can roughly estimate what size 30 elements would be. 
> >>
> >>Is it so that when specifying a XCAP document we also need to 
> >>define what element will be used for indexing when asking for 
> >>a range of elements? 
> >>
> >>Regards
> >>/Henrik 
> >>
> >>
> >>
> >>
> >>>-----Original Message-----
> >>>From: Miguel Garcia [mailto:[email-masked]]
> >>>Sent: den 2 juli 2004 12:14
> >>>To: ext Jonathan Rosenberg
> >>>Cc: Henrik Albertsson (KI/EAB); [email-masked]
> >>>Subject: Re: XCAP Directory: Large response ?
> >>>
> >>>
> >>>Jonathan Rosenberg wrote:
> >>>
> >>>
> >>>>I would propose that we not put this functionality in xcap 
> >>>
> >>>itself, but 
> >>>
> >>>>use the existing HTTP mechanisms. I will note that these 
> >>>
> >>>mechanisms are 
> >>>
> >>>>*byte* oriented, and not XML oriented. Thus, you may cut an 
> >>>
> >>>element off 
> >>>
> >>>>in the middle by asking for the first 50 bytes. However, if 
> >>>
> >>>your main 
> >>>
> >>>>concern is bandwidth or latency, then bytes are much more 
> >>>
> >>>significant of 
> >>>
> >>>>a measure than elements. If I say, "give me 30 elements", 
> >>>
> >>>that could be 
> >>>
> >>>>30 bytes or 30MB, depennding on the amount of data in 
> >>
> >>each element.
> >>
> >>>>-Jonathan R.
> >>>>
> >>>
> >>>Jonathan, I think the HTTP byte ranges will not work. Let's say I 
> >>>request a long document, and I request bytes 1 to 10000. The 
> >>>HTTP server 
> >>>will cut the XML document exacltly when the number of bytes 
> >>>has reached 
> >>>the count, and will send it to the XCAP client. The XCAP 
> >>
> >>client can't 
> >>
> >>>check if the document is well-formed, not even check if it 
> is valid 
> >>>according to the XML schema. As a consequence, it won't 
> >>
> >>display it to 
> >>
> >>>the user, or dare to do any operation on it. The only thing 
> >>
> >>the XCAP 
> >>
> >>>client can do is to request the next byte range.
> >>>
> >>>I am concerned with the limited amount of memory a device 
> >>
> >>might have, 
> >>
> >>>AND the bandwidth. The latter can be solved with byte 
> >>
> >>ranges, but not 
> >>
> >>>the former.
> >>>
> >>>However, if we provide a mechanism to request ranges of 
> >>
> >>entries in a 
> >>
> >>>list, this will solve the problem: the XCAP server create a 
> >>
> >>valid XML 
> >>
> >>>document that complies with the schema and send it to the 
> >>>XCAP client. 
> >>>The client can request additional ranges once it has freed 
> >>>the memory. 
> >>>The drawback is that we cannot accurately control the 
> bandwidth (we 
> >>>don't know how many bytes will come), but I don't think this 
> >>>is a big issue.
> >>>
> >>>- Miguel
> >>>
> >>>-- 
> >>>Miguel A. Garcia           tel:+358-50-4804586
> >>>Nokia Research Center      Helsinki, Finland
> >>>
> >>
> >>_______________________________________________
> >>Simple mailing list
> >>[email-masked]
> >>https://www1.ietf.org/mailman/listinfo/simple
> >>
> 
> -- 
> Miguel A. Garcia           tel:+358-50-4804586
> Nokia Research Center      Helsinki, Finland
> 

_______________________________________________
Simple mailing list
[email-masked]
https://www1.ietf.org/mailman/listinfo/simple

---

From: Jonathan Rosenberg <[email-masked]>
Date: Fri, 02 Jul 2004 16:17:37 -0400

inline.

George Foti (QA/EMC) wrote:

> Comments in line Miguel.
> 
> 
>> -----Original Message----- From: Miguel Garcia
>> [mailto:[email-masked]] Sent: Friday, July 02, 2004
>> 9:03 AM To: George Foti (QA/EMC) Cc: Henrik Albertsson (KI/EAB);
>> Jonathan Rosenberg; [email-masked] Subject: Re: [Simple] RE: XCAP
>> Directory: Large response ?
>> 
>> 
>> George:
>> 
>> I don't like this solution. It requires the XCAP server to do a
>> "trial and error" of the XML document that will be send to the
>> user. Build an XML document with "n" entries, if the size of the
>> document exceeds X bytes, start reducing the number entries, one by
>> one, until the result is less than X bytes. That's awful.
>> 
> 
> It is a managed complexity, and mostly implementation. With some meta
> data, it can be manageable.

I agree with Miguel; this is really hard. It would require a lot of 
implementation complexity.

>> I believe we have two constraints here: one is the display size of
>> the device and the other is the memory of the device. I believe a 
>> device may want to get a page of entries it can render in the
>> display, then release the memory, and request a new page.
>> 
> 

If you want pagination, I think you need to provide that as something 
quite separate from the byte length constraints. Pagination is a very 
application/content specific thing. Indeed, if the goal were to say 
"give me the first N children of this element", thats something that the 
multiple-element selection can do. I think you'd want to then add a 
Ranges header to protect yourself against byte overload in the case 
where an entry is too large.

XPath provides the ability to do that kind of selection. Something like 
this would work:

list/buddy[position() > N and position() < M]

That would give you the Nth through Mth buddy elements.

XCAP doesnt support that level of complexity in the server. If it did, 
and you wanted protection against memory overload (say, 1k max), you'd 
do something like this:

GET http://server/blah-blah/list/buddy[position() > N and position() < M]
Range: bytes=0-1000

If the response is truncated, then you need to adjust your pagination 
size on the client.

-Jonathan R.

-- 
Jonathan D. Rosenberg, Ph.D.                600 Lanidex Plaza
Chief Technology Officer                    Parsippany, NJ 07054-2711
dynamicsoft
[email-masked]                     FAX:   (973) 952-5050
http://www.jdrosen.net                      PHONE: (973) 952-5000
http://www.dynamicsoft.com

---

From: Miguel Garcia <[email-masked]>
Date: Mon, 05 Jul 2004 10:31:30 +0300

See below.

> George Foti (QA/EMC) wrote:

[snip]

> If the response is truncated, then you need to adjust your pagination 

I think this is going in the right direction now. So let me ask a couple 
of questions to verify that the position() is valid for the purpose we want.

1) Is position already part of XCAP? By reading the XCAP base document I 
guess the answer is yes.

2) I guess if we have a non-flat list, the position element will operate 
  in all the leaves of the XML document. For instance, if we have a 
resource list document, with several <list> elements, each one 
containing a few <entry> elements, the position() predicate executed in 
the <entry> element will consider the <entry> element sequentially ordered.

Let me try to explain better with an example. Consider the following 
resource list XML document.

   <list name="friends" uri="sip:[email-masked]" subscribeable="true">
     <entry name="Bill" uri="sip:[email-masked]">
       <display-name>Bill Doe</display-name>
     </entry>
     <entry name="Johanna" uri="sip:[email-masked]">
       <display-name>Johanna Doe</display-name>
     </entry>
     <list name="close-friends" uri="sip:[email-masked]"
           subscribeable="true">
       <entry name="Joe" uri="sip:[email-masked]">
         <display-name>Joe Smith</display-name>
       </entry>
       <entry name="Nancy" uri="sip:[email-masked]">
         <display-name>Nancy Gross</display-name>
       </entry>
       <entry name="Peter" uri="sip:[email-masked]">
         <display-name>Peter Black</display-name>
       </entry>
       <external>http://www.example.org/xcap/resource-lists/users/a/foo
          </external>
     </list>
   </list>

There are 5 <entry> elements, named Bill, Johanna, Joe, Nancy and Peter. 
Imagine I want entries 2 to 4 (in spite they span across different 
lists). Will the HTTP URI be something like:

GET http://server/blah-blah/resource-lists/users/jack/entry[position()>1 
and position()<5]

3) I believe the HTTP byte ranges is a complement of this feature, a 
kind of emergency life jacket that under normal circumstances will not 
be use. I doubt an XCAP client can do much with an XML document that is 
truncated by the byte ranges: it can just discard the contents or store 
them (if it has enough memory) and request the rest of the document.

- Miguel

---

From: Jonathan Rosenberg <[email-masked]>
Date: Mon, 05 Jul 2004 21:27:52 -0400

inline.

> 1) Is position already part of XCAP? By reading the XCAP base document I 

No, it is not. I would not suggest putting this in the base spec, which 
is already overdue and suffering feature creep. I have suggested, in 
another thread, that we perhaps can have an "xcap advanced operations" 
spec that extended xcap with new features, including multiple 
insertions. This would possibly be a candidate feature for such an 
extension, if the group decided to pursue that.

> the <entry> element will consider the <entry> element sequentially ordered.

No. It would apply only at the specific place in the hierarchy where it 
is placed.

> GET http://server/blah-blah/resource-lists/users/jack/entry[position()>1 

No, this will not work. I would suggest that, if you are worried about 
this, the client would page in each level of the hierarchy at a time.

> them (if it has enough memory) and request the rest of the document.

Right. The problem Ranges solves is if a fixed memory is available in 
the device for the content, you can make sure that there is room to 
store the results. If the results are a truncated document, you would 
need to retry with a much smaller page size.

-Jonathan R.

---

From: Miguel Garcia <[email-masked]>
Date: Tue, 06 Jul 2004 10:15:41 +0300

Summary of the thread:

- I think we have consensus on adding a feature that allows an XCAP 
client to request a range of XML elements in a particular schema.

- It seems that the HTP Range header does not serve for the purpose, 
since it operates on bytes and will most likely truncate XML documents.

- It seems that the XPath position() operator will almost serve, but it 
is not part of the XCAP base specification.

- And we don't want to delay the XCAP base spec by adding more features 
at this stage.

- So this feature can be added as an advanced XCAP feature at a later stage.

So far so good. At least I agree with the above. There is one bit where 
I don't quite agree: it seems that the position() opeator in XPath 
operates on the element level, meaning that if we have a non-flat list, 
the XCAP client will need to request ranges in each of the lists (see 
point 2 below).

I think this is bad, and since we are going to design this extended 
feature, we should make it simple to use.

First, if I have a device that can accommodate x entries, my XCAP client 
would need to request x entries for the first list, get the response 
(say y entries came), then do a new query requesting x-y entries in the 
next list, etc. Alternatively, my XCAP client can request x entries en 
each list, and perhaps I will get n * x entries (n is the number of lists).

Second, this way of working assumes that I know in advance the number of 
lists that are nested. This assumption is false when I have never 
downloaded the XML document (such as when I start using a new device).

In short: I want to have the means in XCAP to indicate: "please send me 
this XML document with range of x to z entries, and send them to me in a 
valid XML document". I don't need to know the structure of the document 
in advance (how many deep nested lists and their names), although I know 
the XML schema of the document.

Is this something we can agree upon?

- Miguel

Jonathan Rosenberg wrote:

> inline.
> 
> Miguel Garcia wrote:
> 
>>> XCAP doesnt support that level of complexity in the server. If it 
>>> did, and you wanted protection against memory overload (say, 1k max), 
>>> you'd do something like this:
>>>
>> I think this is going in the right direction now. So let me ask a 
>> couple of questions to verify that the position() is valid for the 
>> purpose we want.
>>
>> 1) Is position already part of XCAP? By reading the XCAP base document 
>> I guess the answer is yes.
> 
> 
>> 2) I guess if we have a non-flat list, the position element will 
>> operate  in all the leaves of the XML document. For instance, if we 
>> have a resource list document, with several <list> elements, each one 
>> containing a few <entry> elements, the position() predicate executed 
>> in the <entry> element will consider the <entry> element sequentially 
>> ordered.
> 
> 
>> There are 5 <entry> elements, named Bill, Johanna, Joe, Nancy and 
>> Peter. Imagine I want entries 2 to 4 (in spite they span across 
>> different lists). Will the HTTP URI be something like:
>>
>> GET 
>> http://server/blah-blah/resource-lists/users/jack/entry[position()>1 
>> and position()<5]
> 
> 
>> be use. I doubt an XCAP client can do much with an XML document that 
>> is truncated by the byte ranges: it can just discard the contents or 
>> store them (if it has enough memory) and request the rest of the 
>> document.
> 
>

---

From: "Joel M. Halpern" <[email-masked]>
Date: Tue, 06 Jul 2004 11:19:21 -0400

I rather disagree with your summary.
1) I am not at all convinced this is necessary,
1') The way you are defining the problem, I am not at all convinced it is 
achievable.

2) I do not think we have anything like a rough consensus on whether this 
is ended, but rather an interesting discussion on possible requirements and 
possible ways to meet it.  The chairs may have observed a consensus, I suppose.

As far as I can tell, requesting x elements of a certain type, from a 
certain point in the XML document, tells you almost nothing about how much 
information you are getting.  If the elements are leaf elements, they may 
contain large or small amounts of content.  If they are not leaf elements, 
then how many contained elements there are depends upon the schema, whether 
the nested elements are optional and / or may occur multiple times, etc.

And if you try to say "I want X elements, just start counting from the 
starting point." then there is no way that the result can confidently be a 
valid XML document or meaningful portion of a document according to a 
schema.  It will take some interesting work to even make it well-formed XML 
under that constraint.

Now, if we add the filter definition Hisham has suggested, we at least have 
a behaviors that is well defined and can produce meaningful results.  It 
even matches a useful and understandable problem.  However, it is a big 
change in the semantics of HTTP GET as far as I can tell.  It starts 
falling into the trap of using HTTP as a generic transport.  Will it work 
through relays?

It seems to me that byte ranges are already there and already give control 
at a level that corresponds to real limits on real clients.  It may be hard 
to construct what you need from it, but it matches the "I only have Y 
buffer space available" constraint.
If we really need to solve this problem over and above that, I guess I 
would prefer to explore whether the suggestion of Filters on GET is valid 
and workable.

Yours,
Joel M. Halpern

At 10:15 AM 7/6/2004 +0300, Miguel Garcia wrote:
>Summary of the thread:
>
>- I think we have consensus on adding a feature that allows an XCAP client 
>to request a range of XML elements in a particular schema.
>
>- It seems that the HTP Range header does not serve for the purpose, since 
>it operates on bytes and will most likely truncate XML documents.
>
>- It seems that the XPath position() operator will almost serve, but it is 
>not part of the XCAP base specification.
>
>- And we don't want to delay the XCAP base spec by adding more features at 
>this stage.
>
>So far so good. At least I agree with the above. There is one bit where I 
>don't quite agree: it seems that the position() opeator in XPath operates 
>on the element level, meaning that if we have a non-flat list, the XCAP 
>client will need to request ranges in each of the lists (see point 2 below).
>
>would need to request x entries for the first list, get the response (say 
>y entries came), then do a new query requesting x-y entries in the next 
>list, etc. Alternatively, my XCAP client can request x entries en each 
>list, and perhaps I will get n * x entries (n is the number of lists).
>
>valid XML document". I don't need to know the structure of the document in 
>advance (how many deep nested lists and their names), although I know the 
>XML schema of the document.
>
>>spec that extended xcap with new features, including multiple insertions. 
>>This would possibly be a candidate feature for such an extension, if the 
>>group decided to pursue that.
>>
>>>
>>>operate  in all the leaves of the XML document. For instance, if we have 
>>>a resource list document, with several <list> elements, each one 
>>Right. The problem Ranges solves is if a fixed memory is available in the 
>>device for the content, you can make sure that there is room to store the 
>>results. If the results are a truncated document, you would need to retry 
>>with a much smaller page size.
>>-Jonathan R.
>
>--

---

From: Miguel Garcia <[email-masked]>
Date: Wed, 07 Jul 2004 09:40:52 +0300

Inline.

> 1) I am not at all convinced this is necessary,

Well, this is not what some other folks thought. I can recall 4 or 5 
folks arguing that this functionality is needed.

> 1') The way you are defining the problem, I am not at all convinced it 
> is achievable.

Again this is a matter of subject interpretation. Everything is 
achievable if you are smart enough to solve the problem.

> 
> 2) I do not think we have anything like a rough consensus on whether 
> this is ended, but rather an interesting discussion on possible 
> requirements and possible ways to meet it.  The chairs may have observed 
> a consensus, I suppose.

It's not me who should judge whether we have consensus or not. I thought 
we had it, but that's my own interpretation. One has to bear in mind 
that consensus is not unanimity. Some folks tend to forget this slight 
difference.

> certain point in the XML document, tells you almost nothing about how 
> much information you are getting.  If the elements are leaf elements, 
> they may contain large or small amounts of content.  If they are not 
> leaf elements, then how many contained elements there are depends upon 
> the schema, whether the nested elements are optional and / or may occur 
> multiple times, etc.

I know it, we discussed it.

> starting point." then there is no way that the result can confidently be 
> a valid XML document or meaningful portion of a document according to a 
> schema.  It will take some interesting work to even make it well-formed 
> XML under that constraint.

Disagree. I can safely say "please send me X elements in a valid XML 
document". The XML document is valid no matter whether it contains 1 or 
a milillion elements in it. The XCAP server is responsible to create a 
valid XML document according to the schema, including the "appropriate" 
number of elements.

We are doing the same with presence documents, whereby the presence 
server creates an XML document with the appropriate number of tuples and 
extensions, according to the authorizationn rules, policy, filters, etc. 
So if we can do it with a presence server, we can do it with an XCAP 
server.

> 
> Now, if we add the filter definition Hisham has suggested, we at least 
> have a behaviors that is well defined and can produce meaningful 
> results.  It even matches a useful and understandable problem.  However, 
> it is a big change in the semantics of HTTP GET as far as I can tell.  
> It starts falling into the trap of using HTTP as a generic transport.  
> Will it work through relays?

It will work with proxies, although I agree that is extending somehow 
the semantics of the GET method in HTTP. Having said that, I don't see 
Hisham's proposal fulfills the requirements. It assumes that you know in 
advance the XML document, and then you apply a filter. It requires you 
to know the values you are trying to filter. We don't want to filter 
based on values, but based on the number of <entry> elements present in 
the XML document. Therefore, I don't think adding a filter to the GET 
request will solve the problem.

> 
> It seems to me that byte ranges are already there and already give 
> control at a level that corresponds to real limits on real clients.  It 
> may be hard to construct what you need from it, but it matches the "I 
> only have Y buffer space available" constraint.
> would prefer to explore whether the suggestion of Filters on GET is 
> valid and workable.

We already had consensus (not unanimity) that byte ranges is a 
complement to the filter. It does not solve any problem by itself, as 
the XCAP server will truncate the XML document when the byte count is 
reached. Your XCAP client can't do anything with the truncated XML 
document. So byte ranges is just a limit to avoid flooding an XCAP 
client, but will not solve any other problem.

> Joel M. Halpern

Regards,

       Miguel Garcia

>> - It seems that the XPath position() operator will almost serve, but 
>> it is not part of the XCAP base specification.
>>
>> - And we don't want to delay the XCAP base spec by adding more 
>> features at this stage.
>>
>> So far so good. At least I agree with the above. There is one bit 
>> where I don't quite agree: it seems that the position() opeator in 
>> XPath operates on the element level, meaning that if we have a 
>> non-flat list, the XCAP client will need to request ranges in each of 
>> the lists (see point 2 below).
>>
>> First, if I have a device that can accommodate x entries, my XCAP 
>> client would need to request x entries for the first list, get the 
>> response (say y entries came), then do a new query requesting x-y 
>> entries in the next list, etc. Alternatively, my XCAP client can 
>> request x entries en each list, and perhaps I will get n * x entries 
>> (n is the number of lists).
>>
>> Second, this way of working assumes that I know in advance the number 
>> of lists that are nested. This assumption is false when I have never 
>> In short: I want to have the means in XCAP to indicate: "please send 
>> me this XML document with range of x to z entries, and send them to me 
>> in a valid XML document". I don't need to know the structure of the 
>> document in advance (how many deep nested lists and their names), 
>> although I know the XML schema of the document.
>>
>>>>> did, and you wanted protection against memory overload (say, 1k 
>>>>> max), you'd do something like this:
>>>>>
>>>>> If the response is truncated, then you need to adjust your 
>>>>> pagination size on the client.
>>>>>
>>>>> -Jonathan R.
>>>>
>>>>
>>>> 1) Is position already part of XCAP? By reading the XCAP base 
>>>> document I guess the answer is yes.
>>>
>>>
>>> No, it is not. I would not suggest putting this in the base spec, 
>>> which is already overdue and suffering feature creep. I have 
>>> suggested, in another thread, that we perhaps can have an "xcap 
>>> advanced operations" spec that extended xcap with new features, 
>>> including multiple insertions. This would possibly be a candidate 
>>> feature for such an extension, if the group decided to pursue that.
>>>
>>>>
>>>> have a resource list document, with several <list> elements, each 
>>>> one containing a few <entry> elements, the position() predicate 
>>>> executed in the <entry> element will consider the <entry> element 
>>>> sequentially ordered.
>>>
>>>
>>> No. It would apply only at the specific place in the hierarchy where 
>>> it is placed.
>>>
>>>>
>>> No, this will not work. I would suggest that, if you are worried 
>>> about this, the client would page in each level of the hierarchy at a 
>>> time.
>>>
>>>>
>>>> kind of emergency life jacket that under normal circumstances will 
>>>> not be use. I doubt an XCAP client can do much with an XML document 
>>>> that is truncated by the byte ranges: it can just discard the 
>>>> contents or store them (if it has enough memory) and request the 
>>>> rest of the document.
>>>
>>>

---

From: Jonathan Rosenberg <[email-masked]>
Date: Wed, 07 Jul 2004 06:28:42 -0400

inline.

> folks arguing that this functionality is needed.

I think "pagination" is a reasonable requirement. It is definitely a new 
one, in that it has not appeared in our requirements document:
http://www.jdrosen.net/papers/draft-ietf-simple-data-req-03.txt

My definition of pagination is that this is a feature specific to 
resource lists. The feature allows the client to request any number of 
items (which include entries, lists, or entry references) at each level. 
Thus, if a user has a two-level hierarchical list, the user could get 
the first level, and see it on their phone. They can then select one of 
the lists on the first level, and the phone will get pages of that list.

This approach is achievable as well.

>> starting point." then there is no way that the result can confidently 
>> be a valid XML document or meaningful portion of a document according 
>> to a schema.  It will take some interesting work to even make it 
>> well-formed XML under that constraint.
> 
> 

No. This is not possible. Joel is correct - you cannot have an algorithm 
for getting X elements of an arbitrary XML document which results in a 
document that is schema valid. Let me give you an example. Lets say I 
have the following schema:

<?xml version="1.0" encoding="UTF-8"?>
<xs:schema xmlns:xs="http://www.w3.org/2001/XMLSchema" 
elementFormDefault="qualified" attributeFormDefault="unqualified">
  <xs:element name="root">
   <xs:complexType>
    <xs:sequence>
     <xs:element name="el1"/>
     <xs:element name="el2"/>
     <xs:element name="el3"/>
    </xs:sequence>
   </xs:complexType>
  </xs:element>
</xs:schema>

This schema *requires* that the <root> element have three children, el1, 
  el2 and el3. If you ask for a version of this document with no more 
than two elements, it cannot be done in a way that results in a schema 
compliant document.

> So if we can do it with a presence server, we can do it with an XCAP 

This "pagination" concept you are interested in is only meaningful for 
specific XML document formats, where there is a natural place in the 
schema were an arbitrary number of elements can be present. This is NOT 
a general XML, XCAP or HTTP feature.

>> results.  It even matches a useful and understandable problem.  
>> However, it is a big change in the semantics of HTTP GET as far as I 
>> can tell.  It starts falling into the trap of using HTTP as a generic 
>> transport.  Will it work through relays?
> 
> 

As I posted in a separate note, I see no point in including filters in 
the HTTP requests. Hisham - what does this add beyond what the URI can 
already too? I know what it takes away - it takes away our ability to 
use PUT, which will mean that we are back to a custom protocol. That is 
definitely months of additional delay. I see no value in doing this.

-Jonathan R.

---

From: Miguel Garcia <[email-masked]>
Date: Wed, 07 Jul 2004 13:59:36 +0300

Inline as well. My comments are prepended by M.G.

> This approach is achievable as well.

M.G: I agree that the "pagination" requirement is applicable to resource 
lists, but it is also applicable to XCAP directory lists. Actually, the 
whole discussion (see the title of this e-mail) started with the XCAP 
directory lists. In general, the requirement is applicable whenever an 
XML element can be infinitely repeated in an XML document.

> 
>>> And if you try to say "I want X elements, just start counting from 
>>> the starting point." then there is no way that the result can 
>>> confidently be a valid XML document or meaningful portion of a 
>>> document according to a schema.  It will take some interesting work 
>>> to even make it well-formed XML under that constraint.
>>
>>
>>
>> document". The XML document is valid no matter whether it contains 1 
>> or a milillion elements in it. The XCAP server is responsible to 
>> create a valid XML document according to the schema, including the 
>> "appropriate" number of elements.
> 
> 

M.G: Yes, you are right. IF the schema mandates the presence of a number 
of elements, those must be present, and cannot be restricted by a range. 
I am not trying to change that behaviour.

I think the precondition for requesting a range of elements is that the 
elemnt is defined with the fololowing attributes

      minOccurs="0" maxOccurs="unbounded"

This is what happens with the <entry>, <list>, <reference> elements in 
the resource list, and the <entry> element in the XCAP directory list. 
In this way, the schema is always respected independently of the range 
of requested elements.

>> server creates an XML document with the appropriate number of tuples 
>> and extensions, according to the authorizationn rules, policy, 
>> filters, etc. So if we can do it with a presence server, we can do it 
>> with an XCAP server.
> 
> 

M.G: I agree.

>>
>>>
>>> Now, if we add the filter definition Hisham has suggested, we at 
>>> least have a behaviors that is well defined and can produce 
>>> meaningful results.  It even matches a useful and understandable 
>>> problem.  However, it is a big change in the semantics of HTTP GET as 
>>> far as I can tell.  It starts falling into the trap of using HTTP as 
>>> a generic transport.  Will it work through relays?
>>
>>
>>
>> Hisham's proposal fulfills the requirements. It assumes that you know 
>> in advance the XML document, and then you apply a filter. It requires 
>> you to know the values you are trying to filter. We don't want to 
>> filter based on values, but based on the number of <entry> elements 
>> present in the XML document. Therefore, I don't think adding a filter 
>> to the GET request will solve the problem.
> 
> 

M.G: I agree.

---

From: "Joel M. Halpern" <[email-masked]>
Date: Wed, 07 Jul 2004 08:13:20 -0400

This mechanism for synthesizing a subset document happens to work with the 
schemas we are using.  Here are two examples of problems.

Given a schema that said that foo elements can have any number of bar 
element children, followed by a baz element, it could get very difficult to 
synthesize a proper document.
Suppose that the real document had inside the foo 8 bars, followed by the 
required baz.  And you ask for 5 elements starting with foo.
Do you really expect the server to give you a document with a foo 
containing 3 bar and a baz?

Suppose that the schema has more nesting than one would like, and foo1 
elements are required to contain foo2 elements, which are required to 
contain foo3 elements.  And you ask for 2 elements starting at foo1.  There 
is no valid document of two elements (by your counting) that the server can 
return.

And these are not the most complex cases that can.
I do not want to make XCAP into a full database mechanism, but I also do 
not want to overly constrain the schemas it can apply to.

I think that trying to constrain the length of an XML document by the 
number of elements in it is simply incompatible with returning something 
that is schema valid, even according to a sensible fragment schema.

Yours,
Joel M. Halpern

At 09:40 AM 7/7/2004 +0300, Miguel Garcia wrote:
>document". The XML document is valid no matter whether it contains 1 or a 
>milillion elements in it. The XCAP server is responsible to create a valid 
>XML document according to the schema, including the "appropriate" number 
>of elements.
>
>We are doing the same with presence documents, whereby the presence server 
>creates an XML document with the appropriate number of tuples and 
>So if we can do it with a presence server, we can do it with an XCAP server.

---

From: Miguel Garcia <[email-masked]>
Date: Wed, 07 Jul 2004 15:55:21 +0300

Inline.

Joel M. Halpern wrote:

> This mechanism for synthesizing a subset document happens to work with 
> the schemas we are using.  Here are two examples of problems.
> 
> element children, followed by a baz element, it could get very difficult 
> to synthesize a proper document.
> Suppose that the real document had inside the foo 8 bars, followed by 
> the required baz.  And you ask for 5 elements starting with foo.

In the example you are describing, the XCAP client will request 5 bar 
elements, that is, will place a limit in the number of bar elements, but 
not on the baz element, since it is mandatory by the schema (remember, 
the XCAP client knows the schema).

So the XCAP server will build an XML document that contains 5 bar 
elements and 1 baz element, and any other mandated element.

We are doing the same with attributes. If an attribute is mandatory by 
the schema, I don't need to request it, I just request the element.

> contain foo3 elements.  And you ask for 2 elements starting at foo1.  
> There is no valid document of two elements (by your counting) that the 
> server can return.

I indicated in a previous e-mail that the precondition to be able to 
request ranges of elements is that the element is declared with 
attributes  minOccurs="0" maxOccurs="unbounded" in the schema. 
Certainly, this does not match your example, because foo2 elements will 
be declared with minOccurse="1" (you said it is required to contain foo2 
elements, which I interpret as 'at least 1 must be present').

What is clear is that we need to define the rules under which an XCAP 
client can request pagination.

> not want to overly constrain the schemas it can apply to.

It is not a matter of turning XCAP into this or that, it is a matter 
into making XCAP a useful protocol. Without this feature I doubt XCAP 
can be widely deployed in mobile devices.

> that is schema valid, even according to a sensible fragment schema.

I disagree. I believe it is possible. Or do you think that when you read 
e-mail through a web interface you are getting unvaild HTML documents? 
Of course not. So the same logic can be applied to XML documents.

I also believe we need to set the rules for the XCAP client to determine 
when it can request pagination.

- Miguel

---

From: "Joel M. Halpern" <[email-masked]>
Date: Wed, 07 Jul 2004 10:40:54 -0400

There appears to be some (reasonable) desire to get part of an XML 
Configuration, where "part" means not a just selection of identified 
elements but some other subseting.
This appears to be driven by some combination of client memory and user 
presentation resources.

Rather than trying to argue that some solution meets the needs, I would 
like to see a clear and reasonably complete list of requirements.  I can 
believe that there is some requirement that is not met by byte-range for 
example, but I do not know what it is.

Yours,
Joel M. Halpern

---

From: Jonathan Rosenberg <[email-masked]>
Date: Wed, 07 Jul 2004 22:05:58 -0400

Miguel Garcia wrote:

>>
>> This schema *requires* that the <root> element have three children, 
>> el1,  el2 and el3. If you ask for a version of this document with no 
>> more than two elements, it cannot be done in a way that results in a 
>> schema compliant document.
> 
> 

OK, so lets push on this a bit.

Lets say I have a bunch of places where this schema constraint is true. 
If you ask for "20 things", which 20 get returned? Is the 20 distributed 
around the document? Is it based on some kind of DFS walk? How do you 
know whether you will return something meaningful to the user, even if 
its schema compliant? In a hierarchical list, I really think you want to 
separately paginate each layer in the hierarchy, by explicit user 
request. That seems implementable to me, and it also seems to match the 
user experience I would expect.

Anyway, as Joel mentioned, it would be good to better understand the 
requirements here.

-Jonathan R.

---

From: Miguel Garcia <[email-masked]>
Date: Thu, 08 Jul 2004 09:27:31 +0300

Inline.

Jonathan Rosenberg wrote:

>> M.G: Yes, you are right. IF the schema mandates the presence of a 
>> number of elements, those must be present, and cannot be restricted by 
>> a range. I am not trying to change that behaviour.
>>
>> I think the precondition for requesting a range of elements is that 
>> the elemnt is defined with the fololowing attributes
>>

I don't have a special requirement here, so I am fine with your proposal 
to do pagination on a per hierarchical level.

> Anyway, as Joel mentioned, it would be good to better understand the 

Right. So let's try to formulate the requirement. We are looking for a 
mechanism that allows an XCAP client to request fractions of an XML 
document, so that the returned document is valid but need not include 
all the repeatable entries. This allows the XCAP client to operate on 
the XML document (render and manipulate it), release the memory, request 
the next fraction of the same XML document, and start the process again.

Regards,

      Miguel

---

From: Miguel Garcia <[email-masked]>
Date: Thu, 08 Jul 2004 09:37:20 +0300

Joel:

Let me try to give you a concrete example where you can see that the 
byte range does not server for the purpose:

Imagine I want to request this very simple XML document

  <foo>
     <bar>This is an element</bar>
  </foo>

Now, when I request this document, I request a byte range (say, 0 to x 
bytes). The server apply the count on the byte range, and cut the 
document. It returns me the following document:

  <foo>
     <bar>This is a

So my XCAP client will not consider the document even well-formed, not 
even valid. So what can my XCAP client do with this document? I have two 
options:

a) request the remaining of the document. But since I applied a byte 
range, presumably because my XCAP client does not have more available 
memory to request a longer document (otherwise, why did I request a byte 
range in the first place), I cannot store this partial document and 
request the remaining parts (I don't have memory to store the remaining 
parts). So this does not solve the problem.

b) dispose the document in the garbage bin (a.k.a. release the memory), 
and request the complete document. But again we are in the same 
situation as in a), the terminal requested a byte range in the first 
place, so what for if it has to request the remaining document?

Is there any other solution you can envision? I don't, and therefore my 
conclusion is that byte ranges does not serve for the purpose.

Regards,

          Miguel

---

From: "Joel M. Halpern" <[email-masked]>
Date: Thu, 08 Jul 2004 08:47:39 -0400

I am sorry.  I can not understand the case where the client can not repair 
the partial answer, but somehow can get a meaningful answer when it is 
provided as complete XML elements.  (I presume the example you gave was 
over-simpilifed, but in the example, there is no anser that will fit.)

a) If the server sends as much as fits in the "buffer", the client can 
always parse forward, trim backwards, add closing elements as needed, and 
end up with more content than would have been shipped if the server had 
shipped only the elements that it somehow knew would fit.
b) given that the length of the content of the elements is not known, I do 
not understand in what terms an element count limit will help a client cope 
with a resource limitation.
c) The requirement as you state it seems to edge in to the solution.  I 
understand that the client software wants to request part of a document.  I 
understand that to present this to the user the client software will need a 
well-formed XML document.  I do not understand why it follows that the 
partial document that is delivered must be well-formed, since the client 
can work it over any way it chooses.  From the last sentence, you seem to 
be driving towards a specific implementation.
d) When we starting putting on constraint about which portions of a 
document can be requested using this mechanism, and what "schema 
compliance" is supposed to mean about the answer we are crafting a very 
specialized solution, rather than tackling a general problem.  And that 
will make it difficult to build compliant XCAP servers that can cope easily 
with multiple schemas and schema changes.

Yours,
Joel M. Halpern

In a separate message Miguel Garcia wrote:
Right. So let's try to formulate the requirement. We are looking for a 
mechanism that allows an XCAP client to request fractions of an XML 
document, so that the returned document is valid but need not include all 
the repeatable entries. This allows the XCAP client to operate on the XML 
document (render and manipulate it), release the memory, request the next 
fraction of the same XML document, and start the process again.

At 09:37 AM 7/8/2004 +0300, Miguel Garcia wrote:
>Joel:
>
>Let me try to give you a concrete example where you can see that the byte 
>range does not server for the purpose:
>
>and request the complete document. But again we are in the same situation 
>as in a), the terminal requested a byte range in the first place, so what 
>for if it has to request the remaining document?
>

---

From: Jonathan Rosenberg <[email-masked]>
Date: Fri, 09 Jul 2004 05:19:00 -0400

inline.

Joel M. Halpern wrote:

> I am sorry.  I can not understand the case where the client can not 
> repair the partial answer, but somehow can get a meaningful answer when 
> it is provided as complete XML elements.  (I presume the example you 
> gave was over-simpilifed, but in the example, there is no anser that 
> will fit.)
> 
> always parse forward, trim backwards, add closing elements as needed, 
> and end up with more content than would have been shipped if the server 
> had shipped only the elements that it somehow knew would fit.

This still might not yield a schema compliant document though.

> b) given that the length of the content of the elements is not known, I 
> do not understand in what terms an element count limit will help a 
> client cope with a resource limitation.

I think that there are really *two* resources that are limited.

One resource is the memory buffer, which has an absolute maximum. The 
HTTP Range headers are ideal for making sure that you never pull more 
than fits in here. Presumably, this limit is never reached except only 
in very unusual error cases, and then if it happens you'd display an 
error to the user. I get this in my cell phone sometimes when viewing 
websites that are too large.

Another resource is the air interface. Since the screen real estate is 
small, the phone is only capable of rendering a few lines of a buddy 
list to a user at a time (say, 10). So, rather than fetching the whole 
thing, the phone just gets the bits that it is rendering. That is called 
"pagination". The size of each page would be determined by UI 
constraints, and is measured in terms of units of "buddies" or "lists" 
(in the hierarchical case).

In pretty much all cases, I would guess that the size of the data in a 
single "page" of rendered data is way, way, way less than the memory 
capacity of the phone.

These two resource constraints are only loosely coupled; its loosely 
coupled in the sense that, if you werent using pagination, and you tried 
to get the whole document, you might hit the memory limit. Pagination, 
as a result, has the benefit of reducing the amount of memory needed at 
any time, and thus makes it far less likely that you would ever hit the 
memory constraint.

If, for some reason, you hit this memory constraint even while 
paginating, you would probably report this as an error to the user, 
through away the content you received, possibly reduce the pagination 
count, and try again.

I dont think you want to mix the usage of the Range headers with the 
pagination, in that, you would never use the Range headers as a way to 
get parts of the document a bit at a time.

-Jonathan R.

---

From: Miguel Garcia <[email-masked]>
Date: Fri, 09 Jul 2004 12:35:46 +0300

Hi Joel:

I will try to explain once more... Inline.

> had shipped only the elements that it somehow knew would fit.

No, I disagree. The client has to check that the document is well-formed 
and valid according to the schema. If the server truncates the document, 
because it reaches the byte count, then the document is not well formed 
and is not valid according to the schema.

> client cope with a resource limitation.

As I said, the resource limitation is not exclusively related to memory, 
but also to display capabilities, etc. So the client can estimate the 
amount of needed memory per element (something big if you want) and the 
amount of entries it can safely display with a limited buffering. I 
believe there is not safe equation indicating the number of bytes and 
entries the client can accept, but some limit in which the client is 
safe to operate.

> understand that the client software wants to request part of a 
> document.  I understand that to present this to the user the client 
> software will need a well-formed XML document.  I do not understand why 
> it follows that the partial document that is delivered must be 
> well-formed, since the client can work it over any way it chooses.  From 
> the last sentence, you seem to be driving towards a specific 
> implementation.

At least I am not trying to drive towards a specific implementation. But 
  I think you are wrgon with the well-formed XML document. We are using 
XML because implementations can check whether the document is 
well-formed or not, and whether the document is valid according to the 
schema. Still with paginated documents I believe this requirement is in 
place. So the document MUST be well-formed and MUST be valid; the client 
has to check it.

> will make it difficult to build compliant XCAP servers that can cope 
> easily with multiple schemas and schema changes.
 >

I don't quite understand. The whole thing about XCAP is to be able to 
manipulate portions of documents, still being compliant with the schema. 
I don't think that "requesting a portion of a document still being 
compliant with the schema" diverges that much from what XCAP is orinally 
meant to do. I don't think the mechanism is so specific, it should have 
been in place in the first place... but since it isn't, let's work 
together to make it fit smoothly.

Regards,

       Miguel

>> even valid. So what can my XCAP client do with this document? I have 
>> two options:
>>
>> memory to request a longer document (otherwise, why did I request a 
>> byte range in the first place), I cannot store this partial document 
>> and request the remaining parts (I don't have memory to store the 
>> remaining parts). So this does not solve the problem.
>>
>> b) dispose the document in the garbage bin (a.k.a. release the 
>> memory), and request the complete document. But again we are in the 
>> same situation as in a), the terminal requested a byte range in the 
>> first place, so what for if it has to request the remaining document?
>>
>> Is there any other solution you can envision? I don't, and therefore 
>> my conclusion is that byte ranges does not serve for the purpose.
>>
>> Regards,
>>
>>          Miguel
>>
>> Joel M. Halpern wrote:
>>
>>> This appears to be driven by some combination of client memory and 
>>> user presentation resources.
>>> Rather than trying to argue that some solution meets the needs, I 
>>> would like to see a clear and reasonably complete list of 
>>> requirements.  I can believe that there is some requirement that is 
>>> not met by byte-range for example, but I do not know what it is.
>>> Yours,
>>> Joel M. Halpern
>>>

---

From: "Joel M. Halpern" <[email-masked]>
Date: Fri, 09 Jul 2004 09:55:48 -0400

Let me try and restate this, with caveats along the way:

1) One goal is to be able to asked for a specific number of XML elements, 
starting at a specific point in the document, and get a well-formed 
document fragment.
1a) As we have discussed, this can only be done at certain points in a 
schema.  At many points in a schema, this request is either impossible to 
meet, or produces nonsense results, for example when there are more 
mandatory elements than the requested count, or when there are mandatory 
elements later in the set than the requested number of optional elements, or...

2) A related goal is to be able to retrieve information that will fit on 
the screen.  The idea is not to waste air bandwidth (a scarce resource) 
shipping things that can not be displayed.
2a) For any schema element with string or token content, or URI content or 
URI attributes, or ... even if the element is a leaf, the actual amount of 
display resources needed to display a single element is not a constant.  As 
such, requesting 10 elements does not correlate with what fits on the screen.

Hence, I am left to conclude that what is being requested is a very special 
case mechanism that can be effectively used only in limited 
circumstances.  Supporting it on the server will require significant 
intelligence.  Using it on the client will require knowledge about the 
length of content that typically does not exist.

If I have understood the requirements you are aiming at, I am left to 
conclude that this is not something that belongs in XCAP.  It may be 
possible to restrict the scope in the definition.  For example, if the 
scope were only a single repeating XML element which must be a leaf, then 
at least the valid construction problem goes away.  It still would not have 
any relationship to display size.  And it would require extending the 
pattern syntax very heavily.

Yours,
Joel M. Halpern

At 12:35 PM 7/9/2004 +0300, Miguel Garcia wrote:
>>it is provided as complete XML elements.  (I presume the example you gave 
>>was over-simpilifed, but in the example, there is no anser that will fit.)
>>do not understand in what terms an element count limit will help a client 
>>cope with a resource limitation.
>
>entries the client can accept, but some limit in which the client is safe 
>to operate.
>
>>the last sentence, you seem to be driving towards a specific implementation.
>
>At least I am not trying to drive towards a specific implementation. 
>But  I think you are wrgon with the well-formed XML document. We are using 
>XML because implementations can check whether the document is well-formed 
>or not, and whether the document is valid according to the schema. Still 
>with paginated documents I believe this requirement is in place. So the 
>document MUST be well-formed and MUST be valid; the client has to check it.
>
>manipulate portions of documents, still being compliant with the schema. I 
>don't think that "requesting a portion of a document still being compliant 
>with the schema" diverges that much from what XCAP is orinally meant to 
>do. I don't think the mechanism is so specific, it should have been in 
>place in the first place... but since it isn't, let's work together to 
>make it fit smoothly.
>
>>Yours,
>>Joel M. Halpern
>
>Regards,
>
>       Miguel
>
>>
>>>>like to see a clear and reasonably complete list of requirements.  I 
>>>>can believe that there is some requirement that is not met by 
>>>>byte-range for example, but I do not know what it is.
>>>>Yours,
>>>>Joel M. Halpern
>>>>