Subject: Re: [Ietf-message-headers] provisional registration: packaged content over http (5 headers)

From: Graham Klyne <[email-masked]>
Date: Tue, 17 Jan 2012 08:44:16 +0000

[Resending: the original got stuck in moderation, didn't make it to the list]

Hi Richard,

A couple of comments here...

My comments below notwithstanding, I don't think there are any specific problems 
for provisional registration (but I think further discussion would be needed for 
progression to permanent registration).

(By way of full disclosure to other readers, I have been a technical advisor to 
the SWORD project which is presenting these proposals.)

On 22/12/2011 17:31, Richard Jones wrote:
> Packaging
>
> Header field name: Packaging
> Applicable protocol: HTTP
> Status: provisional
> Author/Change controller: Richard Jones c/o UKOLN, University of Bath;
> [email-masked]
> Specification Document:
> http://sword-app.svn.sourceforge.net/viewvc/sword-app/spec/tags/sword-2.0/SWORD001.html?revision=377
> Related Information: http://swordapp.org/sword-v2/sword-v2-specifications/
>
>
> Accept-Packaging
>
> Header field name: Accept-Packaging
> Applicable protocol: HTTP
> Status: provisional
> Author/Change controller: Richard Jones c/o UKOLN, University of Bath;
> [email-masked]
> Specification Document:
> http://sword-app.svn.sourceforge.net/viewvc/sword-app/spec/tags/sword-2.0/SWORD001.html?revision=377
> Related Information: http://swordapp.org/sword-v2/sword-v2-specifications/

[[
The Packaging header applies to resources delivered over HTTP which are 
comprised of component resources, and is for uniquely identifying these well 
structured packaged objects in a similar way that Content-Type does for MIME 
formats.
]]

I think this would be a good opportunity to canvas for information about whether 
any other projects are addressing similar issues w.r.t. conveying information 
about packaging or composite object formats in HTTP.  I'm pretty sure this isn't 
a one-off problem.

Packaging doesn't really fall into the role of a content-type, as it doesn't say 
anything about the nature of purpose of the packaged content.  But it also is 
not really a content transfer encoding, as it may convey application-relevant 
metadata in addition to simply encoding content for transfer.

The nearest I can think of that has been addressed in the IETF is the MHTML work 
from some years ago, which uses multipart/related structures to bundle up the 
content of a web page (http://tools.ietf.org/html/rfc2557).  But that doesn't 
really work in this case, as SWORD and related applications are already using a 
number of alternative formats that don't easily map into a multipart/related or 
similar MIME encapsulation structure.

[[
The Packaging request header SHOULD be used by the client during HTTP POST to 
give information to the server about the packaging format used to construct the 
content being POSTed or PUT. Servers SHOULD use this information to unpack the 
supplied content into its component parts. If the server does not understand the 
package format it MUST either store the content as delivered without unpacking 
or respond with 415 (Unsupported Media Type).
]]

It is not clear from this text that the SHOULD here applies to implementations 
of SWORD.  For the header specification document, I think it would be better to 
avoid such normative claims about its use, which might be read as claiming that 
any HTTP client SHOULD use the header. e.g. just say "The Packaging request 
header may be used by a client ..."

> On-Behalf-Of
>
> Header field name: On-Behalf-Of
> Applicable protocol: HTTP
> Status: provisional
> Author/Change controller: Richard Jones c/o UKOLN, University of Bath;
> [email-masked]
> Specification Document:
> http://sword-app.svn.sourceforge.net/viewvc/sword-app/spec/tags/sword-2.0/SWORD001.html?revision=377
> Related Information: http://swordapp.org/sword-v2/sword-v2-specifications/
>
>
> In-Progress
>
> Header field name: In-Progress
> Applicable protocol: HTTP
> Status: provisional
> Author/Change controller: Richard Jones c/o UKOLN, University of Bath;
> [email-masked]
> Specification Document:
> http://sword-app.svn.sourceforge.net/viewvc/sword-app/spec/tags/sword-2.0/SWORD001.html?revision=377
> Related Information: http://swordapp.org/sword-v2/sword-v2-specifications/

[[
The In-Progress request header MAY be used by the client to inform the server 
that the current content payload is not yet complete in some unspecified way 
during PUT, POST or DELETE. For example, there may by further content packages 
that the client plans to deliver to the server before the full content has been 
delivered, or the client may need to carry out other actions against the server 
before confirming that the server can proceed to fully process the content. 
Exact interpretation of this header is left to the server, so it is necessary 
that server/client pairs will have to have a common understanding of its meaning 
which is beyond the scope of this document.
]]

This feels to me like an abuse of the HTTP protocol - if it modifies the intent 
of the method, that would be wrong, but that's not what I think is intended. 
Rather, it seems to modify the interpretation of the resource identified by the 
request URI, which makes me wonder if the intent might not be better conveyed by 
using a different server-designated URI for the parts.

> Metadata-Relevant
>
> Header field name: Metadata-Relevant
> Applicable protocol: HTTP
> Status: provisional
> Author/Change controller: Richard Jones c/o UKOLN, University of Bath;
> [email-masked]
> Specification Document:
> http://sword-app.svn.sourceforge.net/viewvc/sword-app/spec/tags/sword-2.0/SWORD001.html?revision=377
> Related Information: http://swordapp.org/sword-v2/sword-v2-specifications/

[[
The Metadata-Relevant request header MAY be used by the client to instruct the 
server to (attempt to) extract metadata from the supplied content package, 
during PUT, POST or DELETE. Content packages commonly contain both file content 
and metadata about its contents, and during unpacking servers may process this 
metadata in a way which is meaningful to them. If the content package is being 
supplied to an HTTP resource which is not interested in metadata, then it may be 
that the enclosed information will not be correctly or adequately treated. This 
directive allows the client to indicate to the server that there is metadata 
contained within the package which may be of interest to related resources (for 
example a resource which contains the resource receiving the content), and that 
the server should be free to update those resources accordingly.
]]

Do you *really* mean for this to be applicable to HTTP DELETE operations?

(A bit of Googling - e.g. 
http://www.spenceruresk.com/2011/11/http-delete-requests-that-include-a-body/, 
http://stackoverflow.com/questions/299628/is-an-entity-body-allowed-for-an-http-delete-request 
- suggests that there's no specific prohibition of sending data/metadata with a 
DELETE request, but that any such attempt is unlikely to be handled dependably 
by existing software.  And it's really not clear what it would mean to send 
metadata about a resource that is being deleted.)

Did you consider the option that the relevance of metadata might be conveyed by 
a parameter on the Packaging header field?  This header doesn't seem to have any 
purpose independently of the Packaging header.

#g
--

---

From: Julian Reschke <[email-masked]>
Date: Tue, 17 Jan 2012 09:58:31 +0100

On 2011-12-22 18:31, Richard Jones wrote:
> ...
> [email-masked] <mailto:[email-masked]>
> Specification Document:
> ...

Does this field potentially need I18N? The way it's specified now it is 
stuck with ISO8859-1 with no extension point.

Best regards, Julian

---

From: Julian Reschke <[email-masked]>
Date: Tue, 17 Jan 2012 10:05:25 +0100

On 2012-01-17 09:44, Graham Klyne wrote:
> comprised of component resources, and is for uniquely identifying these
> well structured packaged objects in a similar way that Content-Type does
> for MIME formats.
> ]]
>
> I think this would be a good opportunity to canvas for information about
> whether any other projects are addressing similar issues w.r.t.
> conveying information about packaging or composite object formats in
> HTTP. I'm pretty sure this isn't a one-off problem.

+1

> Packaging doesn't really fall into the role of a content-type, as it
> doesn't say anything about the nature of purpose of the packaged
> content. But it also is not really a content transfer encoding, as it
> may convey application-relevant metadata in addition to simply encoding
> content for transfer.
>
> The nearest I can think of that has been addressed in the IETF is the
> MHTML work from some years ago, which uses multipart/related structures
> to bundle up the content of a web page
> (http://tools.ietf.org/html/rfc2557). But that doesn't really work in
> this case, as SWORD and related applications are already using a number
> of alternative formats that don't easily map into a multipart/related or
> The Packaging request header SHOULD be used by the client during HTTP
> POST to give information to the server about the packaging format used
> to construct the content being POSTed or PUT. Servers SHOULD use this

POST *and* PUT?

> information to unpack the supplied content into its component parts. If
> the server does not understand the package format it MUST either store
> the content as delivered without unpacking or respond with 415
> (Unsupported Media Type).
> ]]

No. 415 is for unsupported media types.

> It is not clear from this text that the SHOULD here applies to
> implementations of SWORD. For the header specification document, I think
> it would be better to avoid such normative claims about its use, which
> might be read as claiming that any HTTP client SHOULD use the header.
> e.g. just say "The Packaging request header may be used by a client ..."

 From the description in the spec it's not clear to me at all how this 
is supposed to work, and why it is needed in addition to the 
Content-Type. I'm sure there's a reason, but it really would be good to 
add more explanations.

Best regards, Julian

---

From: Julian Reschke <[email-masked]>
Date: Tue, 17 Jan 2012 10:08:57 +0100

On 2012-01-17 09:58, Julian Reschke wrote:
>> Related Information:
>> http://swordapp.org/sword-v2/sword-v2-specifications/
>> ...
>

Also, did you consider "From"?

"The From request-header field, if given, SHOULD contain an Internet 
e-mail address for the human user who controls the requesting user 
agent. The address SHOULD be machine-usable, as defined by "mailbox" in 
RFC 822 [9] as updated by RFC 1123 [8]:

     From   = "From" ":" mailbox

An example is:

     From: [email-masked]

This header field MAY be used for logging purposes and as a means for 
identifying the source of invalid or unwanted requests. It SHOULD NOT be 
used as an insecure form of access protection. The interpretation of 
this field is that the request is being performed on behalf of the 
person given, who accepts responsibility for the method performed. In 
particular, robot agents SHOULD include this header so that the person 
responsible for running the robot can be contacted if problems occur on 
the receiving end.

The Internet e-mail address in this field MAY be separate from the 
Internet host which issued the request. For example, when a request is 
passed through a proxy the original issuer's address SHOULD be used.

The client SHOULD NOT send the From header field without the user's 
approval, as it might conflict with the user's privacy interests or 
their site's security policy. It is strongly recommended that the user 
be able to disable, enable, and modify the value of this field at any 
time prior to a request."

Best regards, Julian

---

From: Richard Jones <[email-masked]>
Date: Tue, 17 Jan 2012 14:30:49 +0000

Hi Folks,

Thanks for the feedback; I've got some inline comments to see if I
understand what changes I need to make ...

My comments below notwithstanding, I don't think there are any specific
> problems for provisional registration (but I think further discussion would
> be needed for progression to permanent registration).
>

At this stage we're absolutely only looking for provisional registration;
there are other discussions which need to take place before we can see if
we have the resources to go full standards track with any of this.

> (By way of full disclosure to other readers, I have been a technical
> advisor to the SWORD project which is presenting these proposals.)
>
>
>> http://sword-app.svn.**sourceforge.net/viewvc/sword-**
>> app/spec/tags/sword-2.0/**SWORD001.html?revision=377<http://sword-app.svn.sourceforge.net/viewvc/sword-app/spec/tags/sword-2.0/SWORD001.html?revision=377>
>> Related Information: http://swordapp.org/sword-v2/**
>> sword-v2-specifications/<http://swordapp.org/sword-v2/sword-v2-specifications/>
>>
>>
>> Accept-Packaging
>>
>> http://sword-app.svn.**sourceforge.net/viewvc/sword-**
>> app/spec/tags/sword-2.0/**SWORD001.html?revision=377<http://sword-app.svn.sourceforge.net/viewvc/sword-app/spec/tags/sword-2.0/SWORD001.html?revision=377>
>> Related Information: http://swordapp.org/sword-v2/**
>> sword-v2-specifications/<http://swordapp.org/sword-v2/sword-v2-specifications/>
>>
>
> [[
> whether any other projects are addressing similar issues w.r.t. conveying
> information about packaging or composite object formats in HTTP.  I'm
> pretty sure this isn't a one-off problem.
>
> doesn't say anything about the nature of purpose of the packaged content.
>  But it also is not really a content transfer encoding, as it may convey
> application-relevant metadata in addition to simply encoding content for
> transfer.
>
> MHTML work from some years ago, which uses multipart/related structures to
> bundle up the content of a web page (http://tools.ietf.org/html/**rfc2557<http://tools.ietf.org/html/rfc2557>).
>  But that doesn't really work in this case, as SWORD and related
> applications are already using a number of alternative formats that don't
> easily map into a multipart/related or similar MIME encapsulation structure.
>
> [[
> The Packaging request header SHOULD be used by the client during HTTP POST
> to give information to the server about the packaging format used to
> construct the content being POSTed or PUT. Servers SHOULD use this
> information to unpack the supplied content into its component parts. If the
> server does not understand the package format it MUST either store the
> content as delivered without unpacking or respond with 415 (Unsupported
> Media Type).
> ]]
>
> might be read as claiming that any HTTP client SHOULD use the header. e.g.
> just say "The Packaging request header may be used by a client ..."

Ok, good point.  I wanted to try and spin these as not so SWORD specific
(and then to re-use them in the SWORD protocol).  So just substituting
SHOULD for MAY will be sufficient?

>> http://sword-app.svn.**sourceforge.net/viewvc/sword-**
>> app/spec/tags/sword-2.0/**SWORD001.html?revision=377<http://sword-app.svn.sourceforge.net/viewvc/sword-app/spec/tags/sword-2.0/SWORD001.html?revision=377>
>> Related Information: http://swordapp.org/sword-v2/**
>> sword-v2-specifications/<http://swordapp.org/sword-v2/sword-v2-specifications/>
>>
>>
>> In-Progress
>>
>> http://sword-app.svn.**sourceforge.net/viewvc/sword-**
>> app/spec/tags/sword-2.0/**SWORD001.html?revision=377<http://sword-app.svn.sourceforge.net/viewvc/sword-app/spec/tags/sword-2.0/SWORD001.html?revision=377>
>> Related Information: http://swordapp.org/sword-v2/**
>> sword-v2-specifications/<http://swordapp.org/sword-v2/sword-v2-specifications/>
>>
>
> [[
> The In-Progress request header MAY be used by the client to inform the
> server that the current content payload is not yet complete in some
> unspecified way during PUT, POST or DELETE. For example, there may by
> further content packages that the client plans to deliver to the server
> before the full content has been delivered, or the client may need to carry
> out other actions against the server before confirming that the server can
> proceed to fully process the content. Exact interpretation of this header
> is left to the server, so it is necessary that server/client pairs will
> have to have a common understanding of its meaning which is beyond the
> scope of this document.
> ]]
>
> This feels to me like an abuse of the HTTP protocol - if it modifies the
> intent of the method, that would be wrong, but that's not what I think is
> intended. Rather, it seems to modify the interpretation of the resource
> identified by the request URI, which makes me wonder if the intent might
> not be better conveyed by using a different server-designated URI for the
> parts.

Yeah, this is a tricky one to explain, and clearly needs more work.  Let me
try again here, and then if that makes sense I can fold it back into the
docs:

In-Progress is intended as a guiding hand for the client to use to tell the
server that it should expect more content to be delivered to this web
resource before it can be considered "complete".  The following scenarios
would necessitate its usage:

- you are uploading a number of files via a client in a number of discrete
upload steps.  At each file upload, the file (which may be a package) is
sent to the server, which accepts the deposit.  But you don't want the
server to inject the content into any kind of workflow (e.g. for
re-publication on the web) until you have finished uploading all of the
files, so the In-Progress header tells the server that it should expect
more related content in future requests.  On your final upload either omit
the header or send In-Progress: false to tell the server that it can go
ahead and start its workflows.

- you are uploading content from a number of sources.  Perhaps a content
package from a research management/information system and related research
data from lab equipment.  They both pertain to the same "object" on the
server, but will be delivered an uncertain times and from different
sources.  By using In-Progress, both depositors can tell the server that
more data is coming for that object and it is not yet finished.

We had extensive discussion about how to pitch this, and there was some
suggestion about embedding the In-Progress information into the request
/body/ rather than the headers, but this would require specific treatment
of the package itself, which we wanted to avoid.  I think of it as like a
more decoupled version of Transfer-Encoding: chunked.  So, if we can find a
way to describe this which makes sense and doesn't appear to impact the
other purposes of HTTP, that would be good.  Thoughts?

>> http://sword-app.svn.**sourceforge.net/viewvc/sword-**
>> app/spec/tags/sword-2.0/**SWORD001.html?revision=377<http://sword-app.svn.sourceforge.net/viewvc/sword-app/spec/tags/sword-2.0/SWORD001.html?revision=377>
>> Related Information: http://swordapp.org/sword-v2/**
>> sword-v2-specifications/<http://swordapp.org/sword-v2/sword-v2-specifications/>
>>
>
> [[
> The Metadata-Relevant request header MAY be used by the client to instruct
> the server to (attempt to) extract metadata from the supplied content
> package, during PUT, POST or DELETE. Content packages commonly contain both
> file content and metadata about its contents, and during unpacking servers
> may process this metadata in a way which is meaningful to them. If the
> content package is being supplied to an HTTP resource which is not
> interested in metadata, then it may be that the enclosed information will
> not be correctly or adequately treated. This directive allows the client to
> indicate to the server that there is metadata contained within the package
> which may be of interest to related resources (for example a resource which
> contains the resource receiving the content), and that the server should be
> free to update those resources accordingly.
> ]]
>

No, sorry, good catch; will fix in the next revision.

>
> (A bit of Googling - e.g. http://www.spenceruresk.com/**
> 2011/11/http-delete-requests-**that-include-a-body/<http://www.spenceruresk.com/2011/11/http-delete-requests-that-include-a-body/>,
> http://stackoverflow.com/**questions/299628/is-an-entity-**
> body-allowed-for-an-http-**delete-request<http://stackoverflow.com/questions/299628/is-an-entity-body-allowed-for-an-http-delete-request>- suggests that there's no specific prohibition of sending data/metadata
> with a DELETE request, but that any such attempt is unlikely to be handled
> dependably by existing software.  And it's really not clear what it would
> mean to send metadata about a resource that is being deleted.)
>
> Did you consider the option that the relevance of metadata might be
> conveyed by a parameter on the Packaging header field?  This header doesn't
> seem to have any purpose independently of the Packaging hea

I think there's a risk in combining these, because you could imagine that
there is relevant metadata in, say, a PDF, and if this was part of the
Packaging header you couldn't re-use this header outside of that context.

Cheers,

Richard

---

From: Richard Jones <[email-masked]>
Date: Tue, 17 Jan 2012 14:49:08 +0000

Hi Julian,

>> HTTP. I'm pretty sure this isn't a one-off problem.

Is this something we can do through this or another IETF list?

>> (http://tools.ietf.org/html/**rfc2557<http://tools.ietf.org/html/rfc2557>).
>> But that doesn't really work in

Yes.  The use case is that if you've POSTed a package before, you might
want to replace it, so you could PUT a new package (potentially in a
different format) to the original package URI.

> No. 415 is for unsupported media types.

Ok, interesting; what response should be returned?

> From the description in the spec it's not clear to me at all how this is
> supposed to work, and why it is needed in addition to the Content-Type. I'm
> sure there's a reason, but it really would be good to add more explanations.
>

The main issue is that Content-Type is for the mimetype, which would be
something like application/zip, whereas the Packaging header allows us to
define what is inside the zip; for example, it may by a BagIt, or a METS
package, or such?

Do you imagine a way in which the packaging could be included in the
Content-Type?  We did look into Media Formats, but their general adoption
level seemed quite low, and this approach felt simpler.

Cheers,

Richard

---

From: Julian Reschke <[email-masked]>
Date: Tue, 17 Jan 2012 15:57:15 +0100

On 2012-01-17 15:49, Richard Jones wrote:
> Hi Julian,
>
> On Tue, Jan 17, 2012 at 9:05 AM, Julian Reschke <[email-masked]
> <mailto:[email-masked]>> wrote:
>
>     On 2012-01-17 09 <tel:2012-01-17%2009>:44, Graham Klyne wrote:
>
>         ...
>         [[

I think this is something for apps-discuss.

>         (http://tools.ietf.org/html/ rfc2557
>         <http://tools.ietf.org/html/rfc2557>). But that doesn't really
>         work in

So will the resource at the target URI continue to be the packaged 
variant? If not, you really shouldn't use PUT here.

> Ok, interesting; what response should be returned?

Either 400, or you'd have to define a new status code.

>      >From the description in the spec it's not clear to me at all how
>     this is supposed to work, and why it is needed in addition to the
>     Content-Type. I'm sure there's a reason, but it really would be good
>     to add more explanations.
>
>
> something like application/zip, whereas the Packaging header allows us
> to define what is inside the zip; for example, it may by a BagIt, or a
> METS package, or such?

OK, but how does this affect processing?

> Content-Type?  We did look into Media Formats, but their general
> adoption level seemed quite low, and this approach felt simpler.

Potentially a media type parameter?

Best regards, Julian