Subject: [Simple] XCAP issue 6: selecting elements by text value

From: Jonathan Rosenberg <[email-masked]>
Date: Mon, 10 May 2004 02:28:36 -0400

Currently, XCAP allows you to select an element (for deletion, 
modification, insertion, etc.) by its position amongst other siblings of 
the same parent, or by the value of an attribute. There is no way to 
select an element by its text value.

For example, consider this document:

<foo>
   <bar>Hi</bar>
   <bar>There</bar>
</foo>

There is no way to select the bar element whose content is "There" 
without knowing that this element is the second element under parent 
foo. If the client has not cached the entire document, its not likely to 
know the position.

This has come up as a limitation in modifying large ACL lists in CPCP, 
since each of the entries in the list of ACLs is the same element name, 
and differs only by text content.

This limitation is easily fixed by adding support for the .= selector of 
text value. The following xcap expression would select the bar element 
whose value is "There":

http://server/xcap-root/document/foo/bar[.="There"]

This is a relatively easy change to do, and without it, we introduce a 
serious limitation (need to cache the whole document, or need to 
introduce unique attributes for each element). As such, I propose to add 
this feature.

Do folks agree?

Thanks,
Jonathan R.
-- 
Jonathan D. Rosenberg, Ph.D.                600 Lanidex Plaza
Chief Technology Officer                    Parsippany, NJ 07054-2711
dynamicsoft
[email-masked]                     FAX:   (973) 952-5050
http://www.jdrosen.net                      PHONE: (973) 952-5000
http://www.dynamicsoft.com

_______________________________________________
Simple mailing list
[email-masked]
https://www1.ietf.org/mailman/listinfo/simple

---

From: "Joel M. Halpern" <[email-masked]>
Date: Mon, 10 May 2004 11:46:13 -0400

I find myself somewhat torn on this issue.
 From a pure design point of view, this is a good idea.  In designing the 
schema for some work here, we ended up with a somewhat ugly schema in order 
to get the right information into the attributes for XCAP utilization.
On the other hand, the operation of actually finding the right content is 
dramatically simplified (in our implementation) by using only 
attributes.  It would be significantly more complicated to support this 
content based indexing.

There seem to be two reasonable answers.  One answer is to simply allow 
content reference (as Jonathan's example, or some other syntax.)  The other 
alternative would be to allow this, but to note that this causes complexity 
and to allow application usages to prohibit it.  (This does of course beg 
the question of how a client would know whether such references are 
permitted or not.)  Given that "options" are usually a bad choice, from a 
design perspective I guess we should just allow this, although I am afraid 
that many implementations will find it painful.

Yours,
Joel M. Halpern

At 02:28 AM 5/10/2004 -0400, Jonathan Rosenberg wrote:
>There is no way to select the bar element whose content is "There" without 
>knowing that this element is the second element under parent foo. If the 
>client has not cached the entire document, its not likely to know the position.
>
>serious limitation (need to cache the whole document, or need to introduce 
>unique attributes for each element). As such, I propose to add this feature.
>
>Do folks agree?
>
>Thanks,
>Jonathan R.
>--

---

From: "Joel M. Halpern" <[email-masked]>
Date: Mon, 10 May 2004 11:46:13 -0400

Yours,
Joel M. Halpern

---

From: Jonathan Rosenberg <[email-masked]>
Date: Tue, 11 May 2004 13:12:20 -0400

inline.

Joel M. Halpern wrote:

>  From a pure design point of view, this is a good idea.  In designing 
> the schema for some work here, we ended up with a somewhat ugly schema 
> in order to get the right information into the attributes for XCAP 
> utilization.
> On the other hand, the operation of actually finding the right content 
> is dramatically simplified (in our implementation) by using only 
> content reference (as Jonathan's example, or some other syntax.)  The 
> other alternative would be to allow this, but to note that this causes 
> complexity and to allow application usages to prohibit it.  (This does 
> of course beg the question of how a client would know whether such 
> references are permitted or not.)  Given that "options" are usually a 
> bad choice, from a design perspective I guess we should just allow this, 
> although I am afraid that many implementations will find it painful.

I'm with you on the "options are bad" issue here, and I would strongly 
advise staying away from that path. Thus, we should do, or do not. There 
is no try (quoting Jedi Master Yoda, whose wisdom applies to protocol 
design as it does Jedi arts).

There are two aspects of painful that need to be separated out. There's 
the "its painful since I don't currently do it", and "painful since this 
particular feature is hard". I don't think there is anything 
fundamentally hard about this feature, but I'd welcome thoughts on that.

Thanks,
Jonathan R.

---

From: Jonathan Rosenberg <[email-masked]>
Date: Tue, 11 May 2004 13:12:20 -0400

inline.

Thanks,
Jonathan R.

---

From: "Joel M. Halpern" <[email-masked]>
Date: Tue, 11 May 2004 14:53:50 -0400

The key to the implementation complexity for us is that we are using XCAP 
to access what is essentially a virtual configuration.  We have built our 
schema so that the indexing we support is mirrored into the attributes in 
the XML.  This means that one can traverse the configuration tree using 
attribute checks easily.

However, content checks require that the code actually call mechanism 
specific routines to get the specific content.  This is slower, and not 
indexed effectively.  Also, it requires more specialized code in the 
location selection portion of the system in order to determine where we 
want to be.  Hence, this feature is particularly hard to do, will perform 
less well, and is significantly different implementation from the 
rest.   (Under some circumstances, the content may require a separate RPC 
interaction.  I would not want to have to search all by child components 
that way.)

As a side note, mixed content is normally not recommended.  If we assume 
that mixed content is not being used, then .= selection has very limited 
value on a GET.  (All you could learn is the attributes in node.)  For PUT, 
it means that we can use the content to identify the node, rather than 
requiring that there be a unique attribute.  But of course the unique part 
of the content could have been an attribute if the schema were defined that 
way.

Yours,
Joel M. Halpern

PS: One way of looking at this is that without the .= one can essentially 
treat attrbibutes as the definition of the keying for the search 
structure.  With .=, you have to allow for the case where the user knows 
that something is unique, but the implementation doesn't.

At 01:12 PM 5/11/2004 -0400, Jonathan Rosenberg wrote:
>> in order to get the right information into the attributes for XCAP utilization.
>>complexity and to allow application usages to prohibit it.  (This does of 
>>course beg the question of how a client would know whether such 
>>references are permitted or not.)  Given that "options" are usually a bad 
>>choice, from a design perspective I guess we should just allow this, 
>particular feature is hard". I don't think there is anything fundamentally 
>hard about this feature, but I'd welcome thoughts on that.
>
>Thanks,
>Jonathan R.
>
>
>
>--

---

From: "Joel M. Halpern" <[email-masked]>
Date: Tue, 11 May 2004 14:53:50 -0400

Yours,
Joel M. Halpern

---

From: Jonathan Rosenberg <[email-masked]>
Date: Mon, 17 May 2004 02:42:48 -0400

inline.

Joel M. Halpern wrote:

> The key to the implementation complexity for us is that we are using 
> XCAP to access what is essentially a virtual configuration.  We have 
> built our schema so that the indexing we support is mirrored into the 
> attributes in the XML.  This means that one can traverse the 
> configuration tree using attribute checks easily.
> 
> indexed effectively. 

My point is, it sounds like this is how your implementation currently 
works, as opposed to some fundamental problem with allowing for multiple 
indexes (one for attributes, one for text content)?

  Also, it requires more specialized code in the
> want to be.  Hence, this feature is particularly hard to do, will 
> perform less well, and is significantly different implementation from 
> the rest.   (Under some circumstances, the content may require a 
> separate RPC interaction.  I would not want to have to search all by 
> child components that way.)
> 
> value on a GET.  (All you could learn is the attributes in node.)  For 
> PUT, it means that we can use the content to identify the node, rather 
> than requiring that there be a unique attribute.  But of course the 
> unique part of the content could have been an attribute if the schema 
> were defined that way.

Its all about what kind of constraints we introduce into schema design 
if xcap is desired as a tool to manage documents of that schema. I'm 
worried less about a single document that mixes the two approaches as 
different documents which use different approaches. Since both 
approaches are valid ways to design schemas, and since there doesnt (to 
me at least) seem anything fundamentally hard or different about adding 
text content selection in addition to attribute selection, this seems 
like a reasonable feature.

> 
> Yours,
> Joel M. Halpern
> 
> PS: One way of looking at this is that without the .= one can 
> essentially treat attrbibutes as the definition of the keying for the 
> search structure.  With .=, you have to allow for the case where the 
> user knows that something is unique, but the implementation doesn't.

The implementation knows that either the attributes or the text content 
is unique. Ultimately, both represent text strings that would be used as 
a means to select an element. I just dont see a fundamental difference.

-Jonathan R.

---

From: Jonathan Rosenberg <[email-masked]>
Date: Mon, 17 May 2004 02:42:48 -0400

inline.

-Jonathan R.

---

From: "Joel M. Halpern" <[email-masked]>
Date: Mon, 17 May 2004 08:04:55 -0400

I believe this points to the problem.
Suppose you have a schema where the content of some leaf element is peoples 
human readable names.
If there is some other unique ID attribute, then from a schema point of 
view this is not uniquely identifying information.
However, it may be that the human user happens to ask for information using 
this (.=) key.
And either succeeds or fails.
As XCAP is designed, the access attempt is allowed.
Hence, even though the content is not a unique key, I have to build up 
indexing in case a user uses it for keying.  (And for some cases, it may be 
useful keying.)

If we could restrict the use of .= to only be in the case where the 
application usage definer says that it is okay, then I would be very 
happy.  But I do not know how to create such a restriction in the system we 
are using.

Yours,
Joel M. Halpern

At 02:42 AM 5/17/2004 -0400, Jonathan Rosenberg wrote:
>The implementation knows that either the attributes or the text content is 
>unique. Ultimately, both represent text strings that would be used as a 
>means to select an element. I just dont see a fundamental difference.
>
>-Jonathan R.

---

From: "Joel M. Halpern" <[email-masked]>
Date: Mon, 17 May 2004 08:04:55 -0400

Yours,
Joel M. Halpern

---

From: Jonathan Rosenberg <[email-masked]>
Date: Thu, 20 May 2004 05:47:08 -0400

Joel M. Halpern wrote:

> Suppose you have a schema where the content of some leaf element is 
> peoples human readable names.
> However, it may be that the human user happens to ask for information 
> using this (.=) key.

The attempt would fail in the current spec if the GET resulted in 
matching more than one element.

> indexing in case a user uses it for keying.  (And for some cases, it may 
> be useful keying.)

You can say the same thing about the attribute, though. The *schema* 
can't enforce that the attribute value is a unique key, just as it can't 
for content. I can define a format and declare, in the text, that the 
attributes must have this uniqueness property, or, I can declare in the 
text, that the same is true for the content.

Indeed, I don't think that the application usage needs to spell out, in 
its text definition, that an attribute or some content represents a 
unique key neccesarily. It represents for us, another choice in picking, 
and it may be for some cases that its a useful selector for this 
particular request.

That said, the case I'm worried about is where the "right" schema design 
intends for the content to be the unique key. For example, if I want to 
define a list users in an acl:

<allowed-to-join-conference>
   <user>sip:[email-masked]</user>
   <user>sip:[email-masked]</user>
</allowed-to-join-conference>

Without content-based selection, we either need to force the schema 
designer to never use content as the key (which may result in bad schema 
designs), or else add a second attribute that is nothing more than a 
redundant index:

<allowed-to-join-conference>
   <user id="3">sip:[email-masked]</user>
   <user id="4">sip:[email-masked]</user>
</allowed-to-join-conference>

My concern is that there are potential pitfalls in attempting to 
maintain two indices which are both unique. Its particularly problematic 
in cases where the list is being modified by different users that don't 
necesarily hold the entire content in memory (this is the case for some 
CPCP usages). In that case, you know that the content is unique (or if 
its not, thats OK - you'd just overwrite the user already in the list), 
but the only way to make the attribute unique is to have it equal, or be 
a hash of, the content. That would result in documents that effectively 
look like this:

<allowed-to-join-conference>
   <user id="sip:[email-masked]">sip:[email-masked]</user>
   <user id="sip:[email-masked]">sip:[email-masked]</user>
</allowed-to-join-conference>

which is clearly wasteful. Indeed, you really just wouldnt do this, and 
would be forced to design schemas such that text content never 
represented useful keying information. If it did, you'd have to make it 
an attribute.

And so, thats the crux of it. Do we want to have xcap force this style 
of schema design (content useful for selection is attribute-based only), 
even if it's a bad choice based on schema design practices?

My vote was no, and thus the proposal.

> happy.  But I do not know how to create such a restriction in the system 
> we are using.

We could do that, but I'm not sure what is saved, since you presumably 
would write the code once in either case.

-Jonathan R.

---

From: Jonathan Rosenberg <[email-masked]>
Date: Thu, 20 May 2004 05:47:08 -0400

-Jonathan R.

---

From: "Joel M. Halpern" <[email-masked]>
Date: Thu, 20 May 2004 10:22:10 -0400

I am probably looking at this issue through a limited lens, and may be 
coming to the wrong conclusion.  But I'll try to restate that lens for clarity.

First, let me say that I agree that in some cases the alternatives are ugly.

The situation I am facing is that in the schema I am working with, all the 
keying / indexing I want the user to use is in attributes, and all 
attributes are valid keys.  Therefore, the code has indexing structures for 
using these values.
The content values of the leaf nodes are stored in other parts of the 
system that are hard to get to.
1) This actually produces a pretty elegant structure.  I know what keys the 
user can use (position and attribute) and I can structure things to work 
well with those.  I do not need to worry about keeping index structures for 
things which are not usually unique, just in case they happen to be unique 
in some specific config, and the user wants to use them.  Nor do I have to 
worry about duplicate keys in my keying structure, because my schema has 
key declarations (enforced by my underlying system).  So it makes many 
aspects simpler.
2) If the user can index by content, I have to pull all of the content 
values up into the keying structure so that I can search it effectively if 
the user tries to use it as a key, even if it actually is not unique.

I readily agree that the duplication of values between attribute and 
content is to be avoided.  I agree that the pulling of what ought to be 
values into the attributes is unfortunate.  When doing the schema design, 
there were several places where my co-workers had to remind me that 
specific information was needed for keying, and therefore needed to be in 
attribute, not content.
But supporting searching / indexing on content is a can of worms in its own 
right.

Yours,
Joel

At 05:47 AM 5/20/2004 -0400, Jonathan Rosenberg wrote:
>You can say the same thing about the attribute, though. The *schema* can't 
>enforce that the attribute value is a unique key, just as it can't for 
>content. I can define a format and declare, in the text, that the 
>its text definition, that an attribute or some content represents a unique 
>key neccesarily. It represents for us, another choice in picking, and it 
>may be for some cases that its a useful selector for this particular request.
>
>My concern is that there are potential pitfalls in attempting to maintain 
>two indices which are both unique. Its particularly problematic in cases 
>where the list is being modified by different users that don't necesarily 
>hold the entire content in memory (this is the case for some CPCP usages). 
>In that case, you know that the content is unique (or if its not, thats OK 
>- you'd just overwrite the user already in the list), but the only way to 
>make the attribute unique is to have it equal, or be a hash of, the 
>content. That would result in documents that effectively look like this:
>
>would be forced to design schemas such that text content never represented 
>useful keying information. If it did, you'd have to make it an attribute.
>
>And so, thats the crux of it. Do we want to have xcap force this style of 
>schema design (content useful for selection is attribute-based only), even 
>if it's a bad choice based on schema design practices?

---

From: "Joel M. Halpern" <[email-masked]>
Date: Thu, 20 May 2004 10:22:10 -0400

Yours,
Joel

---

From: Jonathan Rosenberg <[email-masked]>
Date: Sun, 23 May 2004 19:49:19 -0400

THanks for the additional detail. More inline.

Joel M. Halpern wrote:

> The situation I am facing is that in the schema I am working with, all 
> the keying / indexing I want the user to use is in attributes, and all 
> attributes are valid keys.  Therefore, the code has indexing structures 
> for using these values.

Ah, and this is the crux of it.

The assumption in my proposal is that there is nothing special at all 
about text content. In your case, you are storing it elsewhere. That 
makes sense generally for schemas where the text content of a request is 
really big and complex (for example, a document or a paragraph of text). 
   Indeed, with large content like that, its unlikely to ever be useful 
as an index for an xcap URI. So, from that perspective, I understand 
your position.

The problem is that the content of an element need not be big 
complicated text. For example, for access control lists, its a URI, 
which makes perfect sense as an index.

> 1) This actually produces a pretty elegant structure.  I know what keys 
> the user can use (position and attribute) and I can structure things to 
> work well with those.

Sure, but you've already got two keys, and I see no problem with having 
three.

>  I do not need to worry about keeping index 
> structures for things which are not usually unique, just in case they 
> happen to be unique in some specific config, 

Attributes might not be unique either, of course, it depends on the 
schema. Indeed, an attribute could also have a lot of data in it, just 
as much as you might see in the content of an element.

> and the user wants to use 
> them.  Nor do I have to worry about duplicate keys in my keying 
> structure, because my schema has key declarations (enforced by my 
> underlying system).  So it makes many aspects simpler.

Perhaps this is a hole in my understanding of schemas, but, is there a 
way to declare an attribute as a key?

> values up into the keying structure so that I can search it effectively 
> if the user tries to use it as a key, even if it actually is not unique.
> 
> values into the attributes is unfortunate.  When doing the schema 
> design, there were several places where my co-workers had to remind me 
> that specific information was needed for keying, and therefore needed to 
> be in attribute, not content.
> But supporting searching / indexing on content is a can of worms in its 
> own right.

I think the real meta-issue here is whether or not an application usage 
should be allowed to say what represents a key, and what doesn't. If we 
don't allow application usages to say that, then I think we need to 
allow for content to be used as an index, since good schema design might 
really require it. In that case, the underlying code would need to index 
  everything, and leave it to the thing producing the documents to not 
create them in a way which makes that indexing horribly inefficient or 
CPU intensive.

If we do allow application usages to say what represents a key and what 
doesn't, then the server would only need to index stuff which it can 
have a guarantee of uniquess and appropriateness for usage as an index. 
Of course, this would complicate the implementation, in the sense that 
you'd need to have a way to tell it what to index, and what not to, for 
each possible element (assuming you wanted a general purpose xcap 
implementation that supported any application usage, which does not 
appear to be a goal in your case, Joel).

On this meta-issue, I'm on the fence. Other thoughts?

-Jonathan R.

---

From: Jonathan Rosenberg <[email-masked]>
Date: Sun, 23 May 2004 19:49:19 -0400

-Jonathan R.

---

From: "Joel M. Halpern" <[email-masked]>
Date: Mon, 24 May 2004 08:19:01 -0400

There is a simple way for a schema to identify what values are unique.  And 
indicate the scope of that uniqueness.  In fact, there are two such 
ways.  The "unique" element is used to simply declare that something is 
unique.  The"key" element declares that something is unique, and that it 
can be used to validate other elements in the document that reference it 
(which are declared with a "keyref" element.)

If (as I think) the "key" / "unique" elements allow all expressions we 
need, then you have raised the interesting idea that an application usage 
can declare that indexing is only allowed with things that are declared 
with "unique" or "key" expressions.  An XML expert should probably review 
how we are talking about using this.  I have always wondered about what 
the  "unique" element was for.

Yours,
Joel

At 07:49 PM 5/23/2004 -0400, Jonathan Rosenberg wrote:
>>  I do not need to worry about keeping index structures for things which 
>> are not usually unique, just in case they happen to be unique in some 
>> specific config,
>
>schema. Indeed, an attribute could also have a lot of data in it, just as 
>much as you might see in the content of an element.
>
>
>>and the user wants to use them.  Nor do I have to worry about duplicate 
>>keys in my keying structure, because my schema has key declarations 
>>(enforced by my underlying system).  So it makes many aspects simpler.
>
>Perhaps this is a hole in my understanding of schemas, but, is there a way 
>to declare an attribute as a key?
>
>don't allow application usages to say that, then I think we need to allow 
>for content to be used as an index, since good schema design might really 
>require it. In that case, the underlying code would need to 
>index  everything, and leave it to the thing producing the documents to 
>not create them in a way which makes that indexing horribly inefficient or 
>CPU intensive.
>
>doesn't, then the server would only need to index stuff which it can have 
>a guarantee of uniquess and appropriateness for usage as an index. Of 
>course, this would complicate the implementation, in the sense that you'd 
>need to have a way to tell it what to index, and what not to, for each 
>possible element (assuming you wanted a general purpose xcap 
>implementation that supported any application usage, which does not appear 
>to be a goal in your case, Joel).
>

---

From: "Joel M. Halpern" <[email-masked]>
Date: Mon, 24 May 2004 08:19:01 -0400

Yours,
Joel