Subject: [Simple] Strength of XML validity requirements in SIMPLE documents

From: Robert Sparks <[email-masked]>
Date: Tue, 14 Sep 2004 09:54:07 -0500

Folks -

We have several drafts that define XML documents. These drafts
all say something along the lines of

"The simple-filter is an XML document [8] that MUST be well-formed and
    SHOULD be valid."

Why are we specifying SHOULD be valid instead of MUST? I'm asking
this for both simple-filter and for the SIMPLE documents in general.

RjS

_______________________________________________
Simple mailing list
[email-masked]
https://www1.ietf.org/mailman/listinfo/simple

---

From: Jonathan Rosenberg <[email-masked]>
Date: Tue, 14 Sep 2004 14:18:28 -0400

Hmm, its a good question. I think that the vast majority of XML docs
produced in the SIP/SIPPING/SIMPLE groups have the same words, and to be 
honest, I had not
given a ton of thought to MUST vs. SHOULD for valid. I can't really
think of a reason why it would ever be the right thing to produce a
document that is not valid, particularly if the recipient does schema
validation. Certainly I think the actual validation of documents should
never be MUST, excepting rare cases like xcap when its a key part of the
protocol. But, thats a different issue.

-Jonathan R.

-- 
Jonathan D. Rosenberg, Ph.D.                600 Lanidex Plaza
Chief Technology Officer                    Parsippany, NJ 07054-2711
dynamicsoft
[email-masked]                     FAX:   (973) 952-5050
http://www.jdrosen.net                      PHONE: (973) 952-5000
http://www.dynamicsoft.com

---

From: Pekka Pessi <[email-masked]>
Date: Wed, 15 Sep 2004 20:04:11 +0300

Jonathan Rosenberg <[email-masked]> writes:
>Hmm, its a good question. I think that the vast majority of XML
>docs produced in the SIP/SIPPING/SIMPLE groups have the same
>words, and to be honest, I had not given a ton of thought to MUST
>vs. SHOULD for valid. I can't really think of a reason why it
>would ever be the right thing to produce a document that is not
>valid, particularly if the recipient does schema validation.

The presence documents may contain all kind of extensions, which
may be only meaningful for a single application using the presence
service. The presence servers or any intermediate nodes (not to
speak about any innocent watcher) can't really know if the
documents are valid unless they know the schema for all of
those extensions.

Like, if composer puts two tuples from two different sources to
the combined document in the presence server. Both the extensions
use ID attributes, but how could the composer know which
attributes have xs:ID type? The following example may be valid, or
then not, but who is to say without a schema?

<?xml version="1.0" encoding="UTF-8"?>
<presence xmlns="urn:ietf:params:xml:ns:pidf"
     xmlns:x="http://example.com/ex/ten/sion"
     entity="pres:[email-masked]">
  <tuple id="id1"><status><basic>open</basic></status></tuple>
  <x:example-extension id="id1" />
</presence>

--Pekka

---

From: Robert Sparks <[email-masked]>
Date: Wed, 15 Sep 2004 12:39:35 -0500

This makes sense. For work going forward, we should have text
in the document definition drafts that call out that this _kind_ of 
thing
can happen.

In the drafts that use the document, though, we might be able to be
more strict.

For instance, consider filter-format and filter-funct.
Would it be helpful for  filter format to say SHOULD be valid,
but filter-funct say that a filter appearing in the body of a
SUBSCRIBE MUST be valid?

RjS

---

From: Jonathan Rosenberg <[email-masked]>
Date: Wed, 15 Sep 2004 22:57:53 -0400

inline.

> then not, but who is to say without a schema?

It depends on how you want to define "valid". If a document has some 
extensions (say, rpid ones), such a document is still valid according to 
just the pidf schema, since it contains well-formed elements from 
another namespace in a place allowed by the schema.

So, when we say "SHOULD be valid", do we mean that all elements in the 
document should be valid according to their defined schemas, or just 
that the document is valid according to the pidf schema?

-Jonathan R.

---

From: Jari Urpalainen <[email-masked]>
Date: Thu, 16 Sep 2004 10:25:16 +0300

ext Jonathan Rosenberg wrote:

> extensions (say, rpid ones), such a document is still valid according 
> to just the pidf schema, since it contains well-formed elements from 

IMO pidf schema requirement is a "MUST" as content-type 
application/pidf+xml is defined by that schema. How could you use some 
instance document with that content-type if it is not valid ? For 
extensions, instance documents can use schemaLocation attribute, but it 
is only a hint to the validator.
--Jari

---

From: "Pete Cordell" <[email-masked]>
Date: Thu, 16 Sep 2004 13:07:38 +0100

Please see below...
----- Original Message ----- 
From: "Pekka Pessi" <[email-masked]>

>   <x:example-extension id="id1" />

I don't know about the specifics of the presence schema, but presumably the
original presence schema specifies <xs:any namespace='##other'/> at some key
points.  Hence the document is valid against the original schema.
(Mumbo-jumbo is valid in the schema sense as long as it is from another
namespace.)  Put another way, a document can be valid in the schema sense,
but still not be semantically correct.

How validity is established for the extensions is another issue.  This comes
down to versioning.  I think this is an area where the group(s) needs to get
a common understanding on how things should be done.  (Schema is actually
weak in this area!  There are ways, but they don't come for free and they
certainly don't happen by accident!)

Pete.
--
=============================================
Pete Cordell
Tech-Know-Ware
pete(at)tech-know-ware.com
http://www.tech-know-ware.com
http://www.xml2cpp.com - Now Available
+44 1473 635863
=============================================

> >Robert Sparks wrote:
... cut ...

---

From: Jonathan Rosenberg <[email-masked]>
Date: Mon, 20 Sep 2004 01:16:09 -0400

inline.

Jari Urpalainen wrote:

> instance document with that content-type if it is not valid ? 

No, that's not what I'm saying.

I'm saying that a document of type application/pidf+xml could be valid 
against the PIDF schema, but it contains extensions (which are allowed 
in the pidf schema), and those extensions make use of elements defined 
in another schema. However, those elements are not valid according to 
that other schema.

In this case, do we consider this document "valid"?

-Jonathan R.

---

From: Pekka Pessi <[email-masked]>
Date: Mon, 20 Sep 2004 14:35:32 +0300

Jonathan Rosenberg <[email-masked]> writes:
>against the PIDF schema, but it contains extensions (which are allowed in
>the pidf schema), and those extensions make use of elements defined in
>another schema. However, those elements are not valid according to that
>other schema.

>In this case, do we consider this document "valid"?

Term valid has at least three different senses with the XML:

1) 

W3C defines that an XML document is valid when it has a DTD and
the document complies with the constraints set forth in DTD.

That is not what we want. No <!DOCTYPE> for presence, no "MUST be
valid", please.

(When RFC 3863 says that the application/pidf+xml document MUST
be well-formed and SHOULD be valid, I guess it just says that
extensions are allowed, even if they are not shown in the DTD.)

2) 

XML schema specification uses a bit different terminology. Schemas
don't specify documents, but trees (item and all its descendants). 
You can assess validity of a particular tree, for instance, all
the XML stuff within <presence> element. The validator can also
skip certain elements (like the wildcard elements used for
extending pidf). So, validator can return different results: tree
and items within it are fully validated, partially validated, or
not validated at all. Also each, item within the tree are valid or
invalid or their validity is not known.  

I'd say that the XML document is schema-valid if its root element
is fully validated and it and all its descendants are valid. This
is not the official definition, however.

This is unfeasible for the intermediate nodes (or processes), too. 
Unless the node knows the schema (and semantics) of all the
extensions, it cannot assess the validity of composed or filtered
documents. I think filtering and composing is precisely the things
that we want to do in the server nodes.

I can also imagine some applications that would just use existing
presence service, just add their non-standard application-specific
status in the published presence document. Now imagine two of
those applications running on the same node.

3)

Beside validity assessment, XML schema specification has the
concept of "local schema-validity", too:

<http://www.w3.org/TR/xmlschema-1/#section-Overview-of-XML-Schema>

I must admit that I don't fully understand what this
means, but I have a good hunch that it is what most of us
want. ;)

A element or attribute is valid if it satisfies the constraints
set by the XML schema. So, <presence> element is valid, if it
contains entity attribute, zero or more <tuple> elements, then
zero or more <note> elements and after then something wildcarded
stuff (I guess that is anything else but pidf-namespace stuff). I
don't understand how the datatype checking of evil types like
xs:ID or xs:IDREF is done when the "local schema-validity" is
determined. I *guess* that ID/IDREF attribute in tuple is valid if
its value is syntactically correct. Like id="_123" is valid but,
id="123" is not. No check for all id's within a document/item and
all of its descendats is required.

So, if we have a application/pidf+xml document, we can simply say
that it MUST conform to its schema and other requirements set
forth in RFC 3863. If we want to be more verbose, we could say
that the root node in the document MUST be partially or fully
validated and none of the validated nodes MUST NOT be invalid. Or
we could invent our own term and say that documents MUST be
locally-schema-valid with the schema XXX.

--Pekka