Subject: Message fragmentation via message/partial and news-specific header fields

From: Bruce Lilly <[email-masked]>
Date: Wed, 01 Sep 2004 11:33:54 -0400

We had had some discussion about message fragmentation via the MIME media type
message/partial,, and there was some text in earlier drafts, however the
matter is conspicuously absent from the recently-produced usefor-usepro-00
draft.  The rules for message fragmentation and assembly based on standard
header fields are discussed in RFC 2046 as amended by RFC 3798.  There was
some discussion of the matter on the ietf-822 mailing list in June 2003 (q.v.).
I also expect some relevant considerations to be reintroduced into ietf-822
mailing list discussion in the near future.

There are several implications for message fragmentation and reassembly
which appear not to have been adequately considered.  I plan to mention
several of these implications.

First there are some questions that need to be considered:
1. a) Where is message fragmentation expected to occur (e.g. during transport
   between transfer agents when it is recognized that the receiving agent
   has some message size limit), and b) by what criteria (presumably some
   site-dependent size limit, but how is that communicated to that site's
   feeding agents)?
2. Where does reassembly take place?
3. How is multiple fragmentation to be handled? [e.g. a 1GB original
   message might be fragmented into 100 MB pieces at some point, which
   might then be further fragmented into 10 MB pieces, etc.  That means
   that a given article may appear multiple times at various places in
   the network; unfragmented, and broken into various different-sized
   pieces -- the original message and the various pieces each having
   separate, unique message identifiers.)
4. a) During fragmentation, which message header fields are to be copied
   to the enclosing message header, and which are to be preserved in
   the enclosed first part message/partial header?  b) Which fields
   need to be copied to the headers of each of the subsequent message
   fragments?
5. During reassembly, which fields of the first part message header
   are to be retained, which are to be discarded; likewise for the
   fields in the header of the enclosed first part?

Specifically regarding question 5, I expect that Approved, Newsgroups,
Distribution, Path, Followup-To, Expires, Summary, Organization,
Injection-Date, Injector-Info, and User-Agent can be safely copied to
the header of the first part and at least Approved, Newsgroups,
Distribution, Path, Injection-Date, and Expires should be copied to
the headers of subsequent parts.  Copying these fields to the first
part is expected behavior when fragmenting per RFC 2046 as amended by
RFC 3798.  I believe there may be difficulties with Control,
Supersedes, Archive, Posted-And-Mailed, Mail-Copies-To, and possibly
Complaints-To.

---

From: Russ Allbery <[email-masked]>
Date: Thu, 02 Sep 2004 01:27:31 -0700

Bruce Lilly <[email-masked]> writes:

> We had had some discussion about message fragmentation via the MIME
> media type message/partial,, and there was some text in earlier drafts,
> however the matter is conspicuously absent from the recently-produced
> usefor-usepro-00 draft.  The rules for message fragmentation and
> assembly based on standard header fields are discussed in RFC 2046 as
> amended by RFC 3798.  There was some discussion of the matter on the
> ietf-822 mailing list in June 2003 (q.v.).  I also expect some relevant
> considerations to be reintroduced into ietf-822 mailing list discussion
> in the near future.

Is there really any point in talking about this in our drafts?

It seems highly unlikely to me that we're going to get the people who
fragment Usenet messages to adopt MIME to do so, even if it would be the
right thing.  There have been extensive discussions in news.software.nntp
about this and the outcome was quite clear.  Given that, I think this
would mostly be an exercise in protocol design for its own sake, and I'm
not sure that it's worth spending the time and effort on it given the
unlikeliness of widespread use.

I'd rather just not mention it and expend our resources on more pressing
problems, unless there's some significant need to attack this problem and
some reasonable belief that people will adopt a standardized solution in
any significant numbers.

-- 
Russ Allbery ([email-masked])             <http://www.eyrie.org/~eagle/>

---

From: Bruce Lilly <[email-masked]>
Date: Thu, 02 Sep 2004 12:01:32 -0400

> Is there really any point in talking about this in our drafts?

Yes, for the same reason that the issue was addressed in RFC 3798 (and
2298 before it).

> right thing.

1. Message/partial fragmentation has already been used on Usenet.
2. Fragmented messages via message/partial may appear at mail-to-news
   gateways.
3a. Message/partial is, AFAIK, the only standardized method for
   message fragmentation and reassembly.  And since it is part of
   MIME, it is widely applicable (e.g. to HTTP as well as Internet
   Message Format).
3b. If there is some other method of fragmentation/reassembly, the same
   sort of issues need to be discussed

> There have been extensive discussions in news.software.nntp
> about this and the outcome was quite clear.

Clear to whom and from what perspective?  Since the discussion didn't
take place here, can you summarize the discussion or point to an
archived summary?

---

From: Russ Allbery <[email-masked]>
Date: Thu, 02 Sep 2004 10:54:20 -0700

> Yes, for the same reason that the issue was addressed in RFC 3798 (and

What was that reason?  I'm not familiar with either of those RFCs.

>    sort of issues need to be discussed

This is all nice and good and my point is that message fragmentation is
something that's only useful on Usenet for binary messages and 99.99% of
the binary posters are never going to use it based on the *extensive*
discussions that have been had with the authors of that software and with
news administrators on news.software.nntp.

So while this might be useful to someone, it is an extreme edge case and
one with which we have little to no implementation experience, which makes
me doubt in the extreme that this working group is going to be able to do
an adequate job of handling the problem.

>> There have been extensive discussions in news.software.nntp about this
>> and the outcome was quite clear.

> take place here, can you summarize the discussion or point to an

As mentioned above, the outcome of the discussion was a complete and utter
lack of interest in anything MIME-related for message fragmentation,
despite several of us pushing it pretty hard.  What is being used is not
something that we can or should standardize.

---

From: Bill Davidsen <[email-masked]>
Date: Thu, 02 Sep 2004 14:30:36 -0400

> something that we can or should standardize.
Hell, yENC is even standardized in practice, about 20% of all postings I 
am asked to exampne for inappropriate content defy any decode to which I 
have access. If there's one for UNIXen which will handle these posts, 
which don't follow the "standard" at yenc.org, I haven't found it.

People will do what they choose, and to date they choose hacks.

-- 
    -bill davidsen ([email-masked])
"The secret to procrastination is to put things off until the
  last possible moment - but no longer"  -me

---

From: Bruce Lilly <[email-masked]>
Date: Thu, 02 Sep 2004 14:44:49 -0400

> What was that reason?  I'm not familiar with either of those RFCs.

Briefly (this was discussed on ietf-822 in June 2003) the DSN RFC
introduced header fields which need to be treated similarly to
Message-ID w.r.t. first part message header and enclosed message
header for fragmentation and reassembly.  Several of the news-
specific fields will also need to be treated differently than
what is the case per RFC 2046 rules in lieu of any amendments
to those rules.  To take one particular example, the Control
field par RFC 2046 rules would have to be copied to the first
part message header in order to survive fragmentation/reassembly
(only specific fields in the enclosed message header from the
original message header which is encapsulated in the first
part are copied during reassembly).  That means that that first
part will be interpreted as if it were a (complete) control
message.  If RFC 2046 rules were to be amended (as they were
for DSN-specific fields by RFCs 2298/3798) then the Control
field could remain encapsulated within the first part, not
copied to the first part message header, the partial messages
would not be treated as control messages (only the reassembled
whole would be treated as such).

> something that's only useful on Usenet for binary messages

No, it's useful for any message whose size exceeds some threshold,
including large text documents (such as, oh, maybe
draft-ietf-usefor-article-13.txt for example).

> and 99.99% of
> the binary posters are never going to use it

If the functionality is built into UAs and/or injection agents
and/or transport agents is can be transparent to the users. Especially
bearing in mind the fact of common mail/news UAs not to mention
web browser/messaging UAs -- MIME message/partial applies to mail,
news, and HTTP, so support for message/partial by any one component
of such common-use UAs becomes available to the others; provided
the news-specific issues are properly addressed.

> based on the *extensive*

Exactly which "that software" are you talking about -- some
NNTP implementation?

> an adequate job of handling the problem.

This group is likely the only group which can determine how its
news-specific header fields should be handled during fragmentation
and reassembly.

> despite several of us pushing it pretty hard.

If the discussion was only in a narrow special-interest area for
NNTP software, it likely missed UA authors as well as implementors
of transport methods other than NNTP.

>  What is being used is not

Then we should stick to the implications for message/partial.

---

From: Russ Allbery <[email-masked]>
Date: Fri, 03 Sep 2004 11:11:46 -0700

Bruce Lilly <[email-masked]> writes:

> Message-ID w.r.t. first part message header and enclosed message header
> for fragmentation and reassembly.  Several of the news-specific fields
> will also need to be treated differently than what is the case per RFC
> 2046 rules in lieu of any amendments to those rules.  To take one
> particular example, the Control field par RFC 2046 rules would have to
> be copied to the first part message header in order to survive
> fragmentation/reassembly (only specific fields in the enclosed message
> header from the original message header which is encapsulated in the
> first part are copied during reassembly).  That means that that first
> part will be interpreted as if it were a (complete) control message.  If
> RFC 2046 rules were to be amended (as they were for DSN-specific fields
> by RFCs 2298/3798) then the Control field could remain encapsulated
> within the first part, not copied to the first part message header, the
> partial messages would not be treated as control messages (only the
> reassembled whole would be treated as such).

Ah, I see.  Yes, that makes sense.  I don't personally find work on
message/partial important enough to spend time on it, but if you come up
with language to deal with these issues, I'd be willing to review it and
say whether I think it makes sense.

> draft-ietf-usefor-article-13.txt for example).

Splitting text messages that small is obnoxious and anti-social and is
something that I'd filter out on my news server.  General consensus in
news.software.nntp among news administrators (this came up as well) pretty
much matched that feeling.  Nothing smaller than 1MB should be split on
Usenet, and preferrably as far as I'm concerned nothing smaller than 10MB
(although 1MB is the more common metric).

>> and 99.99% of the binary posters are never going to use it

> If the functionality is built into UAs and/or injection agents and/or
> transport agents is can be transparent to the users.

Yes, I know.  The authors of the many different software packages used in
binary newsgroups, which is by and large the only place that news
administrators want to ever see fragmented posts, are uninterested in
implementing it.  This was literally a two year long discussion, involving
a much larger number of people than have ever participated in this working
group.  You can see the conclusion in effect on Usenet right now, namely
widespread implementation of yEnc.

>> based on the *extensive* discussions that have been had with the
>> authors of that software and with news administrators on
>> news.software.nntp.

> Exactly which "that software" are you talking about -- some NNTP
> implementation?

No, software used in binary newsgroups (for posting, serving, and
reading).

> If the discussion was only in a narrow special-interest area for NNTP
> software, it likely missed UA authors as well as implementors of
> transport methods other than NNTP.

Quite possibly.  It won't make any difference, but hey, until I actually
took part in the two year long discussion, I didn't believe it either.

You can trust me or you can tilt at the windmill, your choice.  Given how
long I spent tilting at the windmill, I'm certainly not going to begrudge
anyone else the opportunity.

---

From: "Charles Lindsey" <[email-masked]>
Date: Tue, 14 Sep 2004 13:41:32 GMT

>say whether I think it makes sense.

Well it would be stupid to split a control message using message/partial,
but in the event that it did happen, it would be better to have the
Control-header in the first part (I don't care whether it is also included
in the encapsulated article, though it would be illogical not to do so).

The point is that most agents receiving the message, and particularly
serving agents which are the ones that would implement any control action,
are most unlikely to possess the capability of reassembling the various
parts, or even to have any concept of message/partial at all. Therefore it
is essential that they see the first part as an ordinary control message
and act upon it accordingly. The subsequent parts would then appear in the
relevant groups as ordinary articles, which might be confusing but would
not be likely to do any harm.

In the event that some serving agent did have reassembly capability, then
either

1) It would notice the Control-header first and act upon it. It should
then be smart enough not to bother with reassembly or, if it did
reassemble, to know that it had already done the Control stuff. But even
if it tried to act upon the Control message twice, that would not actually
cause any harm. Or:

2) It would notice the Content-Type: message partial first, in which case
it would then reassemble the message and insert it back into the system,
in which case it would be acted upon as a control message.

>(although 1MB is the more common metric).

And indeed there is a NOTE in draft-13 which makes that point. I don't
think it has found its way into the new drafts yet, but it should.

-- 
Charles H. Lindsey ---------At Home, doing my own thing------------------------
Tel: +44 161 436 6131 Fax: +44 161 436 6133   Web: http://www.cs.man.ac.uk/~chl
Email: [email-masked]      Snail: 5 Clerewood Ave, CHEADLE, SK8 3JU, U.K.
PGP: 2C15F1A9      Fingerprint: 73 6D C2 51 93 A0 01 E7 65 E8 64 7E 14 A4 AB A5

---

From: "Charles Lindsey" <[email-masked]>
Date: Tue, 28 Sep 2004 18:50:57 GMT

>Supersedes, Archive, Posted-And-Mailed, Mail-Copies-To, and possibly

Having thought about this some more, I have concluded that nearly all
headers about which RFC 2046 says nothing specific should be copied to the
headers of the first fragment (and mostly to the subsequent fragments
too), because the reassembly process will then put them back in the
article proper upon reassembly. The only ones which need to be placed in
the enclosed header in the first fragment (in addition to those mentioned
in RFC 2046) are those that might otherwise have been different in the
various fragments. Lines is an obvious candidate here, and also
User-Agent (since injectors are allowed to add bits to it). I see no
reason to make exception for any others (and hence the changes we need to
make to RFC 2046 are rather slight).