Subject: Feedback on proposal

From: Steve Walker <[email-masked]>
Date: Thu, 23 Jun 2005 20:26:06 -0500

I just stumbled across your website/mailing list and thought I 
would share my first impression.  First of all, I've developed 
Usenet software and ran commercial news servers for many years, 
so I understand the protocols well.  While I haven't read the 
proposal 100% yet, what I have read very disappointing.  From 
what I can tell, your work group is going to do more damage to 
Usenet than good.  What is being proposed isn't Usenet at all. 
It's an entirely new protocol and to call it Usenet will just 
confuse people.  At least call it UseneXT so people won't try to 
use a real Usenet client/server with it.

I agree the old RFC's needed some modernization, but instead of 
working with the good and fixing the bad you are proposing a 
whole new, incompatible, protocol.  Usenet was designed around 
the ideas of using the minimal amount of unnecessary controls. 
It is this limited chaos which has allowed it to grow and adapt 
into what it is today.

Writing books of unnecessary and incompatible rules is going to 
fragment the software base.  On one hand you will have the people 
that read RFC's like holy bibles and on the other you will have 
people that say, I've been doing it like this for 15 years and 
not going to change.

Making rules that correspond with Generally Accepted Usenet 
Practices (GAUP) is a good thing.  Trying to force rules that 
break GAUP will divide the Usenet community and software 
developers.

For example, the GAUP on Message-ID length is less than 250 
bytes.  Formalizing this is RFC's is good.  The software 
developers can then use a fixed size buffers and know that if the 
message-id doesn't fit they should drop it anyway.

Another example, requiring the use of MIME/base64 on Usenet 
articles is against decades of practice.  As a matter, of fact we 
filter any base64 encoded article that arrives.  Base64 is 
considered the encoding method of clueless and spamers on Usenet.

Requiring a strict format of the Path header is pointless and 
places extra CPU load on the servers.  The same thing can be said 
by the dozens of nit-picking rules I saw.

Usenet has always been about requiring the very minimum and 
accepting everything.  You should be making rules that simplify 
software, not cause overly complected, buggy software to be 
developed.

As a news server operator I already have to deal with hundreds of 
news clients.  They can't get the existing protocol right, let 
alone follow the hundreds of new, pointless, rules in this 
proposal.  In the real world, the client creators will not 
support this proposal as it's too complex. It just won't happen 
and there is no need for them to.  The existing protocol works 
well, it just needs some minor fixes that simply life, not 
complicate it.

As for the servers I run, I have no plans on making any efforts 
to make changes as specified in these proposals.  That is, unless 
it becomes GAUP.  If you are writing this to see your name on 
electronic paper, then you will succeed.  If you are writing this 
to make a positive difference, I don't see that happening.  Take 
a step back and ask yourself, "when this is all done and all the 
server owners have spent hundreds of millions of dollars 
upgrading software, what has really changed?"  We still have a 
discussion system that grows faster than hardware capability.  We 
still have a million articles of spam per day.  We still have 
problems with article number synchronization on distributed 
systems. We still have stupid kids dumping 100 DVD images a day 
into a misspelled newsgroup that no one will read.

Why don't you use your efforts to solve real problems, instead of 
creating new ones.  If you are going to design an incompatible 
protocol at least fix something major in the original.

Steve Walker
News Admin
NNTPServer.com

---

From: Russ Allbery <[email-masked]>
Date: Thu, 23 Jun 2005 19:49:18 -0700

Steve Walker <[email-masked]> writes:

> Another example, requiring the use of MIME/base64 on Usenet articles is
> against decades of practice.  As a matter, of fact we filter any base64
> encoded article that arrives.  Base64 is considered the encoding method
> of clueless and spamers on Usenet.

I stopped reading at this point.  I used to spend lots of time educating
people who didn't understand the difference between MIME and base64, but
this is no longer a new concept.  I've stopped.  At this point,
particularly if someone is going to claim to be some authority on how
Usenet works, they should already understand this.

If you had investigated a random newsgroup on your server before posting,
you would discover that MIME messages are already the majority.  I picked
rec.arts.poems at random, and out of 7,132 messages, 4,317 use MIME (61%).
In other words, they're already existing best practice, and increasingly
so.

Please check the rest of your objections against reality; you will find
that many of them are similarly mistaken or based on misunderstandings of
how existing software works.

-- 
Russ Allbery ([email-masked])             <http://www.eyrie.org/~eagle/>

---

From: Steve Walker <[email-masked]>
Date: Fri, 24 Jun 2005 00:59:00 -0500

> Usenet works, they should already understand this.

While I do not pretend to be a mime expert, I do know what mime 
is.  I'm fully aware of the fact that you can have a mime message 
without using base64.  I'm also aware of the fact that most 
base64 encoded messages are also mime and they are not welcome on 
Usenet.  Remember the problems when AOL released their mime 
enabled, base64 encoding, users onto Usenet several years back.

> In other words, they're already existing best practice, and increasingly

I installed a filter to count mime messages.  Of the 75,000 
articles tested so far, mime messages account for about 2% of 
Usenet articles.

This proposal is trying to declare the 98% of usenet articles are 
invalid. I can't even imagine why anyone would even consider 
trying this.  If you really like mime, suggest that people use 
it, but don't attempt to require the impossible.

> Please check the rest of your objections against reality; you will find

I'm bringing up real world problems.  You can't claim to be the 
"standard" when 98% of the world is against it.

Steve Walker
News Admin
NNTPServer.com

---

From: Russ Allbery <[email-masked]>
Date: Fri, 24 Jun 2005 06:09:11 -0700

Steve Walker <[email-masked]> writes:

> While I do not pretend to be a mime expert, I do know what mime is.  I'm
> fully aware of the fact that you can have a mime message without using
> base64.  I'm also aware of the fact that most base64 encoded messages
> are also mime and they are not welcome on Usenet.

Sorry, this is simply not true as universally as you're claiming.

> I installed a filter to count mime messages.  Of the 75,000 articles
> tested so far, mime messages account for about 2% of Usenet articles.

I find this incredibly unlikely.  How are you counting MIME messages?

---

From: Henry Spencer <[email-masked]>
Date: Fri, 24 Jun 2005 11:11:59 -0400 (EDT)

On Fri, 24 Jun 2005, Steve Walker wrote:
> ...I'm fully aware of the fact that you can have a mime message 
> base64 encoded messages are also mime...

However, the reverse is not true:  most MIME messages do *not* use base64
encoding.  (Hint:  every message in this mailing list is a MIME message.)

> ...mime messages account for about 2% of Usenet articles.
> invalid.

What on Earth are you talking about?  There is nothing in our drafts that
makes a non-MIME article invalid.  Use of MIME -- MIME, not base64 -- is
mandatory only when non-ASCII characters are involved, which is precisely
when it *should* be mandatory, if only to specify what character set is
being used.

                                                          Henry Spencer
                                                       [email-masked]

---

From: Bruce Lilly <[email-masked]>
Date: Fri, 24 Jun 2005 11:51:44 -0400

> I find this incredibly unlikely.  How are you counting MIME messages?

I agree with Russ.  Note that any message containing a MIME-Version
header field (N.B. "mImE-VeRsIoN" is case-insensitive) is by definition
[RFC 2045] a MIME message; RFC 2045:

    (1)   A MIME-Version header field, which uses a version
          number to declare a message to be conformant with MIME
          and allows mail processing agents to distinguish
          between such messages and those generated by older or
          non-conformant software, which are presumed to lack
          such a field.

Any message containing any message header or MIME-part header field
beginning with "Content-" (case-insensitive) is also apparently intended
to be a MIME message, with one possible but highly unlikely exception
(details below); again quoting RFC 2045:

  MIME-extension-field := <Any RFC 822 header field which
                           begins with the string
                           "Content-">

  MIME-message-headers := entity-headers
                          fields
                          version CRLF
                          ; The ordering of the header
                          ; fields implied by this BNF
                          ; definition should be ignored.

  MIME-part-headers := entity-headers
                       [fields]
                       ; Any field not beginning with
                       ; "content-" can have no defined
                       ; meaning and may be ignored.
                       ; The ordering of the header
                       ; fields implied by this BNF
                       ; definition should be ignored.

Note that:
1. Any message containing any MIME extension field (i.e. beginning
   with "content") is supposed to contain a MIME-Version field in
   the message header
2. A MIME-Version field may not appear in a MIME-part header (it
   may of course appear in the message header of an encapsulated
  "message" composite media type).

One or both of those rules are sometimes broken by non-conforming
software.  One particular large vendor of broken software is notorious
for breaking the second rule.

Messages can be categorized as:
o MIME -- contains a MIME-Version field in the message header
o broken -- contains any Content- field but lacks a MIME-Version
  message header field (but see below re. Content-Type), or
  contains a MIME-Version field in a MIME-part header
o non-MIME -- no MIME-Version field (anywhere) and no fields
  (anywhere) beginning with case-insensitive "Content-", with
  the possible exception of Content-Type (in the absence of a
  MIME-Version message header field, a message with a [RFC 1049]
  Content-Type field and no other Content- fields is a non-MIME
  message).

---

From: Henry Spencer <[email-masked]>
Date: Fri, 24 Jun 2005 12:18:42 -0400 (EDT)

On Fri, 24 Jun 2005, Bruce Lilly wrote:
>   ...the possible exception of Content-Type (in the absence of a

Although (as Bruce noted) this is pretty unlikely:  RFC 1049 was fairly
obscure and not widely implemented.  It's technically possible for a
non-MIME message to have Content-Type, but in practice, it's a pretty safe
bet that the presence of Content-Type means it's really a MIME message but
produced by broken software.

                                                          Henry Spencer
                                                       [email-masked]

---

From: Bruce Lilly <[email-masked]>
Date: Fri, 24 Jun 2005 13:16:06 -0400

> produced by broken software.

Yes, and in the unlikely event that such a message is found, the RFC 1049
syntax is sufficiently different from MIME syntax to be able to determine
which is the case.

---

From: Steve Walker <[email-masked]>
Date: Fri, 24 Jun 2005 12:55:18 -0500

> when it *should* be mandatory, if only to specify what character set is

Usefor:
Section 2.3 MIME Conformance is mandating that agent MUST support 
mime.

So any client that doesn't post mime can no longer be called an 
NNTP client?  You just killed hundreds of news clients.

The spec says on one hand the minimum amount of headers is xxx, 
but also says that mime is required, but not all required mime 
headers are required Usenet headers.  This is a conflict.

MIME doesn't deal well with existing Usenet practices.  yEncoding 
and MIME don't mix well.  Considering a very large percentage of 
Usenet articles are yEncoded you are creating a situation that 
has no resolution.  They will not quit posting just because 
 >someone< says they >have< to use use mime/base64.

Usepro wrote:
NOTE: this corresponds to the range of octets permitted for MIME
         "8bit data" [RFC 2045].  Thus raw binary data cannot be
         transmitted in an article body except by the use of a
         Content-Transfer-Encoding such as base64.

This completely breaks a very large percentage of articles.

I started monitor the Usenet stream for 'yEnc' in the Subject 
header which is required for yEncoding.  I'm seeing (at least) 
65% of all Usenet articles are yEnc.  Some yEnc'ed articles do 
not use the required yEnc in the subject line.  These are not 
being counted in my numbers.

The monitoring for mime is showing slightly higher mime counts 
during the day, it's climbed up to the 3-4 percent mark.

So the argument is still valid: this proposal is trying to force 
clients to use mime regardless of the fact that very few posts 
are currently mime and it would directly break the majority of 
all Usenet posts.

Steve.

---

From: Steve Walker <[email-masked]>
Date: Fri, 24 Jun 2005 13:01:38 -0500

> I find this incredibly unlikely.  How are you counting MIME messages?

I'm counting any article with a MIME-Version: header.  Isn't this 
header required for mime?  If you have a better suggestion I'd be 
happy to adjust the filter and provide new numbers.

Steve.

---

From: Steve Walker <[email-masked]>
Date: Fri, 24 Jun 2005 13:09:54 -0500

> [RFC 2045] a MIME message; RFC 2045:

Good.  Then my filter is checking in the correct way.  It 
searches for the 'mime-version' header (case-insensitive) and 
counts them vs the number of articles that due not have that header.

About 96% of Usenet articles do not have the 'mime-version' 
header.  To require something that less than 4 percent is 
currently doing is not wise.  You will not get the other 96% to 
switch to mime just because this "standard" is published.  The 
standard is non-mime by GAUP and certainly non-base64.

Steve.

---

From: Sebastian Brocks <[email-masked]>
Date: Fri, 24 Jun 2005 20:11:46 +0200

Steve Walker schrieb:

> You will not get the other 96% to 
> switch to mime just because this "standard" is published.

The vast majority of that 96% is binary traffic, which does not
care about standards anyway.

---

From: Steve Walker <[email-masked]>
Date: Fri, 24 Jun 2005 13:38:02 -0500

> care about standards anyway.

So you write something knowing no one will follow and call it the 
standard.  Why don't we write a "standard" that follows standard 
practices.

The software developers can, and do, follow standards.  Otherwise 
they wouldn't be able to read/post at all.  The problem is when 
we get 100 people each doing the same thing 100 different ways. 
This is exactly what a "standard" should solve.  Provide a set of 
guidelines that everyone can use.  To make a bunch of rules, 
knowing that the majority of people can't follow, even if they 
wanted to, is dumb.

If you have plans on ignoring the needs of the single largest 
group of Usenet users, the binary posters and readers, you should 
specify that your rules only apply to the small-x text-only groups.

This should be a "standard" covering "Usenet", all of Usenet, not 
just the tiny portion [insert person here] reads.

Steve.

---

From: Henry Spencer <[email-masked]>
Date: Fri, 24 Jun 2005 14:37:53 -0400 (EDT)

On Fri, 24 Jun 2005, Steve Walker wrote:
> > makes a non-MIME article invalid...
> 

Correct.  The user-interface *software* must know about MIME, because MIME
articles might appear (which is already true) and the user may want to
read them or prepare followups to them (again already true).  That doesn't
mean the software has to use MIME headers in every article it posts.  They
are mandatory only when things like non-ASCII characters show up in the
articles.

> but also says that mime is required...

It says that MIME *support* is required in the *software*.  That's quite
different from saying that MIME headers are required in the articles.

> Usenet articles are yEncoded...

How odd that I read hundreds of Usenet articles a day -- and have been
doing so for 24 years -- and I've never (that I recall) encountered one. 
You might want to produce some numbers to support that "very large
percentage" part.  Bear in mind that "Usenet" and "news" are two different
things; not all news is Usenet. 

> 65% of all Usenet articles are yEnc...

I think you should be saying "news" and not "Usenet"; I would conjecture
that those articles are heavily concentrated in hierarchies that are not
part of Usenet proper.  Indeed, it would be interesting to know just where
all that yEnc traffic is. 

I've generally ignored the yEnc business and will have to leave it to
others to comment on it.  I will note that there's nothing in our spec
that says that news software can't have features beyond what we describe. 

> clients to use mime...

No, this proposal requires that clients *understand* MIME, so they can
handle reading of, and followup to, MIME articles -- which by your own
figures, are already present in modest but significant numbers.  There
is no requirement that they use it in articles they originate.

> ...and it would directly break the majority of all Usenet posts.

No.  See above.

                                                          Henry Spencer
                                                       [email-masked]

---

From: Steve Walker <[email-masked]>
Date: Fri, 24 Jun 2005 14:17:46 -0500

> doing so for 24 years -- and I've never (that I recall) encountered one.

So you want to write a "standard" based only on what you 
personally use it for?  Usenet users are a very large and diverse 
group. One person or a small group of people can't possibly know 
how everyone else uses it.

> things; not all news is Usenet. 

Care to provide the RFC that explains that groups that you don't 
read are no longer part of the Usenet news system?

And I did provide numbers.  I'm running the entire Usenet news 
feed through a filter counting yEncoded articles.  It's showing 
about 65 percent of all articles are yEncoded.

> all that yEnc traffic is. 

Mainly, alt.* of course.  Are you claiming that this standard 
does not apply to articles posted to alt.* groups?  If so, it 
should state that.

I've been running servers for a long time, and I know that alt.* 
groups are without a doubt the most popular groups.  They get 
more posts and they generate more reads than all other groups 
combined. To ignore them when designing a "standard" is unthinkable.

> that says that news software can't have features beyond what we describe. 

It says that binary data must be mime/base64 encoded.  That 
breaks the yEnc standard and 65% of all Usenet posts.

Steve.

---

From: Bruce Lilly <[email-masked]>
Date: Fri, 24 Jun 2005 15:34:10 -0400

On Fri June 24 2005 14:01, Steve Walker wrote:

> header required for mime?

Are you checking in a case-insensitive manner? are you including
Microsoft-generated messages that erroneously put the field in a
MIME-part header instead of the message header?

---

From: Bruce Lilly <[email-masked]>
Date: Fri, 24 Jun 2005 15:36:12 -0400

On Fri June 24 2005 14:09, Steve Walker wrote:

> currently doing is not wise.

Nothing in the specification requires all messages to be MIME
(as noted earlier by Henry).  You may be misinterpreting the
specification.  I';; send a detailed reply to your longer
message in a while.

---

From: Bruce Lilly <[email-masked]>
Date: Fri, 24 Jun 2005 16:07:46 -0400

> It says that binary data must be mime/base64 encoded.

It would help if you would cite the specific draft and section so that
others can clearly see what specific text you are referring to.

You also might wish to carefully read surrounding text to make sure it
doesn't mention things like "future extensions", etc.

You are of course welcome to provide *constructive* criticism; you might
want to read the following RFCs and references first, though:
   [BCP9]     Bradner, S., "The Internet Standards Process --
              Revision 3", BCP 9, RFC 2026, October 1996.

   [BCP14]    Bradner, S., "Key words for use in RFCs to Indicate
              Requirement Levels", BCP 14, RFC 2119, March 1997.

   [FYI17]    Harris, S., "The Tao of IETF - A Novice's Guide to
              the Internet Engineering Task Force", RFC 3160,
              August 2001.

   [BCP25]    Bradner, S., "IETF Working Group Guidelines and
              Procedures", BCP 25, RFC 2418, September 1998.

   [BCP83]    Rose, M., "A Practice for Revoking Posting Rights to IETF
              Mailing Lists", BCP 83, RFC 3683, March 2004.

   [BCP94]    Wasserman, M., "Updates to RFC 2418 Regarding the
              Management of IETF Mailing Lists", BCP 94, RFC 3934,
              October 2004.

   [BCP95]    Alvestrand, H., "A Mission Statement for the IETF",
              BCP 95, RFC 3935, October 2004.

   [RFC1796]  Huitema, C., Postel, J., and S. Crocker, "Not All
              RFCs are Standards", RFC 1796, April 1995.

   [RFC1958]  Carpenter, B., "Architectural Principles of the
              Internet", RFC 1958, June 1996.

   [RFC3426]  Floyd, S., "General Architectural and Policy
              Considerations", RFC 3426, November 2002.

   [RFC3439]  Bush, R. and D. Meyer, "Some Internet Architectural
              Guidelines and Philosophy", RFC 3439, December 2002.

   [RFC3444]  Pras, A. and J. Schoenwaelder, "On the Difference
              between Information Models and Data Models",
              RFC 3444, January 2003.

   [Errata]   RFC-Editor errata page,
              http://www.rfc-editor.org/errata.html

and of course the primary normative references for our work, RFCs
2045-2049, 2231, 2234, and 2822.

> That  

"yEnc" is not a standard.  If you believe otherwise, then (after carefully
reading RFC 1796) please cite the full Standard RFC in which it is specified.

Note that nothing precludes a proper specification of an encoding scheme
similar to "yEnc", and in fact there have been two Internet-Drafts (one
written by Ned Freed, the other by Dan Kohn) which have attempted to do so.
See
http://www.imc.org/ietf-usefor/2003/Feb/0506.html
http://www.imc.org/ietf-usefor/2003/Feb/0575.html
http://www.imc.org/ietf-usefor/2003/Feb/0577.html

---

From: Steve Walker <[email-masked]>
Date: Fri, 24 Jun 2005 15:23:26 -0500

> specification.  I';; send a detailed reply to your longer

Thanks for pointing this out.  I will go back and read more 
carefully.  In the real world, if all clients must support mime 
how do non-mime articles get posted.  I really don't see software 
developers who design clients that allow non-mime posting to 
bother with fully supporting reading mime.  Why require this in 
the "standard", why not leave it up to the clients.  We'll end up 
with a much of half hearted attempts at supporting the very 
complex mime system. This will greatly complicate the programing 
needed to generated a standards compliant client.  Simple is better.

Regardless, it does prohibit binary data in the message body 
unless it is MIME encoded which will break a large number of 
articles.

Steve.

---

From: Bruce Lilly <[email-masked]>
Date: Fri, 24 Jun 2005 16:21:25 -0400

On Fri June 24 2005 13:55, Steve Walker wrote:

> MIME doesn't deal well with existing Usenet practices.

You might want to have a look at the USEFOR WG charter:
http://www.ietf.org/html.charters/usefor-charter.html

Note that we're working on "best current practice" and "best
proposed practice".  The word "best" is not an accident.  Just
because something has been done (leading in many cases to
interoperability problems) does not mean that it's a good idea.

> yEncoding  
> and MIME don't mix well.

No, see the two proposed drafts mentioned separately.

> Considering a very large percentage of  
> has no resolution.

We didn't create the non-standard "yEnc" situation.  Nobody
prevented a proposal for such an encoding scheme within the
existing messaging framework, and at least two people have
separately tried to fix that oversight.

> They will not quit posting just because  

Nobody is saying that.
 
>          Content-Transfer-Encoding such as base64.

You understand the meaning of "such as", right?

> header which is required for yEncoding.

"Required" by which Standards Track RFC?  (By the way, you might
want to read RFC 4096.  And definitely read RFC 2822 section 3.6.5
paying special attention to the word "unstructured" (defined in
2822 sect. 2.2.1))

---

From: Steve Walker <[email-masked]>
Date: Fri, 24 Jun 2005 15:37:08 -0500

> reading RFC 1796) please cite the full Standard RFC in which it is specified.

With 65% of articles using yEnc and less than 4% using mime, I'd 
say it more of a real standard than mime.  Just because the ietf 
doesn't call it a standard doesn't make it an industry standard 
due to widespread use.  If you wrote a "standard" that goes 
against common practice who do you think will win?  It's better 
to write a standard using common practice as the baseline.

What is the point to write a new standard that no one will use it?

Instead of trying to break existing practices, why doesn't anyone 
address any of the real problems facing the Usenet news system?

For example, why doesn't the "standard" specify that 64-bit 
integers are used for article counters?  A real-world problem 
facing NNTP developers is what happens when the 31/32 bit counter 
rolls over?  I highly doubt that most clients can deal with this 
issue.  Some clients are going to roll at 2^31 and some at 2^32.

Changes like this and others would actually provide some benefit 
to making a new standard.

Steve.

---

From: Ned Freed <[email-masked]>
Date: Fri, 24 Jun 2005 14:36:59 -0700 (PDT)

> No, see the two proposed drafts mentioned separately.

FWIW, I would be happy to revive the draft if there is any interest.
The last time I tried this the silence was fairly deafening, however.

I should also mention that Dan and I saw no problem, and indeed some benefit,
from combining the two drafts.

				Ned

---

From: Harald Tveit Alvestrand <[email-masked]>
Date: Fri, 24 Jun 2005 23:42:47 +0200

Steve,

thanks for reading the proposals before commenting.....

I'm sure you'll appreciate that this group has been around and trying to 
write standards for about 7 years, and has been through a massive amount of 
discussion about just about every topic related to USENET. We're hoping to 
finish the format description document real soon now - as I'm sure you can 
see from the traffic, there are some outstanding issues.

Asking the group at this time to reevaluate early basic decisions such as 
the decision to not depart from the RFC 2822 message format, and to 
recommend MIME tagging of messages, is certainly not exactly optimal; while 
you have only recently discovered the group, most of the participants have 
already spent considerable cycles figuring out what should and should not 
be done.

You've raised an issue - that binary postings common in alt. groups will 
apparently not be conformant with the new format description. If you can 
figure out exactly which part of the USEFOR draft that creates this 
conflict, and can propose language to deal with that, I'm sure we can 
follow up on that.

But please don't come in and say "You're all wrong, you need to start all 
over". We're not at that stage of the group's lifecycle.

                        Harald Alvestrand
                         USEFOR WG chair

--On 23. juni 2005 20:26 -0500 Steve Walker <[email-masked]> wrote:

>
> I just stumbled across your website/mailing list and thought I would
> share my first impression.  First of all, I've developed Usenet software
> and ran commercial news servers for many years, so I understand the
> protocols well.  While I haven't read the proposal 100% yet, what I have
> read very disappointing.  From what I can tell, your work group is going
> to do more damage to Usenet than good.

---

From: Bruce Lilly <[email-masked]>
Date: Fri, 24 Jun 2005 17:55:58 -0400

On Fri June 24 2005 16:23, Steve Walker wrote:

> In the real world, if all clients must support mime 
> how do non-mime articles get posted.

The same way as always. A plain text message in US-ASCII is just
fine as a non-MIME message.

Incidentally, whether you know it or not, your Mozilla Thunderbird
client already supports MIME (modulo bugs):

Message-ID: <[email-masked]>
Date: Fri, 24 Jun 2005 15:23:26 -0500
From: Steve Walker <[email-masked]>
User-Agent: Mozilla Thunderbird 1.0.2-1.3.3 (X11/20050513)
X-Accept-Language: en-us, en
MIME-Version: 1.0

If you read a MIME message, you might not be aware that it is
MIME, unless you're using a non-MIME client.  For example,
you might not realize that the subject of this message is
MIME-encoded (RFC 2047/2231) unless you take special steps
to look at the raw message content.  Likewise for the body text.
MIME is the only current method which provides for standards-
conforming non-ascii content with language tagging for text (see
RFC 2277).  Maybe that doesn't concern you personally, but there
is a world full of people with names such as Claus André Färber
who would like to have their names displayed properly.
 
> I really don't see software  
> bother with fully supporting reading mime.

They don't have to.  Nobody forces conformance to any Internet
specification; it's purely voluntary.  And separate packages
such as metamail provide MIME reading capability to old-fashioned
non-MIME clients.

> Why require this in  
> the "standard", why not leave it up to the clients.

Interoperability.

> We'll end up  
> complex mime system.

As opposed to the current situation, which is...worse.

> This will greatly complicate the programing  

Life is complex.

---

From: Russ Allbery <[email-masked]>
Date: Fri, 24 Jun 2005 21:35:12 -0700

> I should also mention that Dan and I saw no problem, and indeed some
> benefit, from combining the two drafts.

It's probably worth noting that the binary Usenet is essentially an
extended P2P network at this point and really bears little and decreasing
relationship to text Usenet or any other application of the protocol.
After a lot of discussions on news.software.nntp about this, it's not
completely clear to me that the binary Usenet is even well-served by using
the same article format as everything else.  Their needs and their
standard usage patterns are almost completely different.  (For example,
you could drop threading completely from the binary Usenet and people
would hardly even notice.)

Now that I've realized that this thread was started by someone who pretty
much cares only about the binary groups, this all makes much more sense.
I would expect reactions like this from most people who think that the
binary groups are the primary point of Usenet, even though they're being
carried on a fast-decreasing number of servers and have become somewhat of
a speciality pay market.

---

From: Steve Walker <[email-masked]>
Date: Sat, 25 Jun 2005 08:44:43 -0500

> relationship to text Usenet or any other application of the protocol.

Usenet was designed to allow people to share information.  It is 
the original p2p network.  What people share isn't important.  It 
doesn't matter if it's a sci-fi story, a letter of protest, or a 
picture of a train.  It's people sharing information.  The 
protocol shouldn't tailored for a small group, but should be 
broad and flexible.

> the same article format as everything else.  Their needs and their

A good protocol will be flexible to allow any kind of 
information.  It appears to me that what has been written so far 
is just for the minority.

My point is, that if it's agreed that mime is a good format then 
the word 'SHOULD' be used and not 'MUST".  The same thing for 
yEncoded articles.  Why make hard absolute rules that can not be 
followed by a large number of users.

> much cares only about the binary groups, this all makes much more sense.

That certainly not true.  I care about all of Usenet.  This 
proposal only seems to only care about a small subset of users. 
You are writing rules the will certainly break existing 
practices. I have not suggested anything that would harm any 
group of users.  Why make mime processing required and not 
recommended?  Why forbid non-encoded binary data?  It seems to me 
that neither of these things are harming Usenet right now and 
they are not likely to actually change because of a new "standard".

I'd also like to point out that no one person here has commented 
on the suggestion of requiring article numbers to be 64 bit. 
Here we have a real problem that is bound to cause problems in 
the near future, yet since it won't effect text users from a long 
time it's ignored.

While it may seem that I'm the binary user advocate, it's just 
because from what I've read, It's a group of users that have been 
completely ignored by this proposal.  I just don't want to see 
further seperation between user bases.

Steve.

---

From: Bruce Lilly <[email-masked]>
Date: Sat, 25 Jun 2005 09:59:00 -0400

> The last time I tried this the silence was fairly deafening, however.

It probably makes sense, though as noted last time, there may be some
benefit to decoupling compression and encoding binary->8-bit:
o some binary content, e.g. http://users.erols.com/blilly/mparse/tm35
  body content, won't compress well and may in fact be expanded by gzip
  "compression".
o combining compression and encoding can yield benefits even for 8-bit
  content encoded to a 7-bit transport. E.g. given a 1 MB 8-bit text
  source, compression will typically yield approx. 500 kB binary
  compressed data.  Base64 encoding will expand that by about 26:19
  (4/3 * 78/76) yielding about 684 kB which is obviously smaller than
  what encoding alone would yield.
o similarly, binary content could be compressed and encoded for 7bit
  transport, yielding a size reduction compared to encoding alone.
o Continuing along the same lines, large 7-bit content could be
  compressed and encoded for 7-bit transport to yield an overall size
  reduction, e.g. if one part of the transport path is known to be a
  low-bandwidth or size-constrained bottleneck (typically 1 MB -> 684 kB
  as detailed above).
o compression could be used even where binary transport is available
  (though it's a stretch to call that an "encoding") without the need
  to hide the nature of the original media type by wrapping the compressed
  data in a label identifying the data as "application/x-gzip" or
  "application/octet-stream".

I found out yesterday that 8BITMIME works much more widely than expected,
so this probably should be handled separately from the USEFOR WG (which
is already behind schedule with its work items).  Accordingly, I've
copied the ietf-822 list and set Reply-To to point there.

> from combining the two drafts.

If you think I can be of assistance, feel free to contact me off-list(s).

---

From: Bruce Lilly <[email-masked]>
Date: Sat, 25 Jun 2005 10:35:24 -0400

On Sat June 25 2005 09:44, Steve Walker wrote:

> the word 'SHOULD' be used and not 'MUST".

"MUST" applies to implementations that claim conformance to the
specification.

> The same thing for  
> yEncoded articles.

Unfortunately, there's no stable specification to refer to.  No
IETF community review.  Probably no adequate analysis for security
issues.

> Why make hard absolute rules that can not be  

As separately mentioned, packages such as metamail make MIME
available.  And there are no "hard absolute rules"; implementations
not claiming conformance are free to do as they please.  Of course,
they might not interoperate well with fully conforming implementations.
 
> You are writing rules the will certainly break existing 

Harmful practices should be deprecated, and in some cases prohibited
(by implementations claiming conformance).

> I have not suggested anything that would harm any  
> group of users.

Frankly, you haven't suggested much that's specific enough to
comment on.

> Why make mime processing required and not  
> recommended?

That's "support", not "processing".  And "required" only for
implementations that wish to claim conformance to the specification.
The reason is to ensure interoperability, e.g. for internationalization
of text in header fields.

> Why forbid non-encoded binary data?

It's not backward compatible with the message format which has been used
since RFC 850.  It's not backward compatible with widely deployed transport
and access mechanisms (NNTP, IMAP) which are compatible with and based on
the existing specifications (RFCs 1036 and 822 among others).

> It seems to me   
> that neither of these things are harming Usenet right now

This WG has carefully considered both issues and determined otherwise.

> and  

Maybe, maybe not.  There is no enforcement of Internet specifications
other than peer pressure based on interoperability considerations.
Our task is to produce a specification that takes backward compatibility
with standards-conforming deployed implementations and *best* current
practices into account.
 
> the near future, yet since it won't effect text users from a long 

I believe that you are mistaken.  Our current priority is the "USEFOR"
specification, and it currently provides for article identifiers (in
the Xref header field) as:

   xref            =  "Xref:" SP [FWS] server-name
                      1*( FWS location ) [FWS] CRLF

   server-name     =  path-identity

   location        =  newsgroup-name ":" article-locator

   article-locator =  1*( %x21-27 / %x29-3A / %x3C-7E )
                      ; US-ASCII printable characters
                      ; except '(' and ';'

      NOTE: The traditional form of an article-locator (as required by
      [NNTP]) is a decimal number, with articles in each newsgroup
      numbered consecutively starting from 1.

RFC 2234 ABNF specifies what the "1*" signifies.  Briefly, it means
that the length of the "article-locator" has *no* upper bound (in
practice there is an upper bound of 996 characters minus the length of
the relevant newsgroup name due to line length limits).  That's
significantly larger that what can be represented in a 64-bit integer.
Implementations are free to use whatever mechanisms they deem appropriate
to meet the specification.  An integer representation limited to merely
64-bit numbers (you neglected to state whether that's signed or unsigned,
BTW) might not be a good design choice...  So now you have a comment on
that particular issue.

---

From: Russ Allbery <[email-masked]>
Date: Sat, 25 Jun 2005 09:39:36 -0700

Steve Walker <[email-masked]> writes:

> A good protocol will be flexible to allow any kind of information.

Textual Usenet requires a significant amount of metadata, and handling
that data in an interoperable fashion requires structures that may well be
unnecessary for binary posts.  You're objecting, in part, to the addition
of rules that are *required* in order to do internationalization properly,
and that, as I have already demonstrated, are already used in the majority
of posts to text Usenet (and growing).  In other words, the data that
you're presenting is evidence of exactly my point; the schism between the
needs of the binary Usenet and the text Usenet is deep and growing fast.

See also much of the news.software.nntp discussion over the past two
years, including the discussions of yEnc, multipart binary reassembly, and
the observation that message IDs in their current form really don't make a
lot of sense for binary traffic.

I know that the binary folks don't want to use MIME.  The text folks have
to if we're going to get anywhere with interoperable non-English support.
The standard doesn't require MIME; that's about as much of a concession as
I think you'll ever get out of the people, like me, who care about text.

> It appears to me that what has been written so far is just for the
> minority.

That rather depends on how you measure the minority.  Mostly-text or
only-text is the *vast* majority of Usenet installations; I bet there are
under 100 servers in the entire world that are actually trying to carry a
complete binary feed, and that number similarly is dropping due to the
bandwidth and specialized management requirements.  The significant
majority of Usenet sites at this point (and I'm fairly confident of this
statement -- as a server implementor, I talk to a lot of them) are
carrying either only a handful of binary groups for legacy reasons or no
binary groups at all.

The majority of news readers, both by number and by deployment count, find
dealing with MIME much easier than dealing with yEnc.  For occasional
binary attachments to otherwise text groups (such as for private
hierarchies, mailing list gateways, and then like), MIME is a
significantly better protocol and is considerably more interoperable than
either yEnc or uuencode.

The needs for binary groups are different.

I don't particularly want to *break* binary Usenet, but the problems that
I care about fixing are text problems.  If you want to champion things
that binary Usenet needs, that's great, but you're going to need to not
stomp on the things that text needs in the process.

> My point is, that if it's agreed that mime is a good format then the
> word 'SHOULD' be used and not 'MUST".

As other people have already pointed out, this is the case already.

News *readers* have to support MIME because they have to be able to read
MIME messages, as there will be quite a lot of them.  This means that pure
binary download clients will likely not be compliant *news readers*
according to this standard.  I'm not sure if this is a serious enough
problem to worry about; they're not really trying to be news readers in
the sense that we use in these documents.

> The same thing for yEncoded articles.

I believe yEnc is so poorly designed as to be unstandardizeable by the
IETF in its current form.  I came to that conclusion after spending a
bunch of time trying to prove the opposite.  If you can do better, please,
by all means, go for it.  Just because something is widely used doesn't
mean that it's documented well enough that one can standardize it.  In
practice, there are serious interoperability problems between different
yEnc implementations.

Certainly, given the widespread use of yEnc, it would be worth at least
documenting the existing protocol in an informational RFC.  If you're
willing to do that work, I certainly won't stand in your way.  The
designer appears to have no interest in working with a standards body.

>> Now that I've realized that this thread was started by someone who
>> pretty much cares only about the binary groups, this all makes much
>> more sense.

> That certainly not true.  I care about all of Usenet.

I'm sorry.  That wasn't the right thing for me to say.  I don't know what
you care about, and shouldn't be talking as if I did.  The concerns that
you presented are primarily binary-related concerns.

> Why make mime processing required and not recommended?  Why forbid
> non-encoded binary data?

Because the point of standards is to get different implementations to
interoperate properly with each other, and both of those constraints are
required in order to do so.  Note that not even yEnc is unencoded binary
data.  It *is* a different CTE, and really should be registered as a CTE,
but there's a limit to what a standards body can do if the people
designing protocols refuse to standardize them.

You *have* to encode binary data before putting it on Usenet because the
existing Usenet software is not clean in nul handling and relies on line
termination for finding the end of the article.

> I'd also like to point out that no one person here has commented on the
> suggestion of requiring article numbers to be 64 bit.

That's because you're in the wrong working group.  This is an NNTP issue,
and you're about three years late to raise it (even apart from the fact
that we talked about this and there are reasons that I won't go into for
why it wasn't changed).  This working group doesn't deal with article
numbers.

> Here we have a real problem that is bound to cause problems in the near
> future, yet since it won't effect text users from a long time it's
> ignored.

It wasn't ignored.

> While it may seem that I'm the binary user advocate, it's just because
> from what I've read, It's a group of users that have been completely
> ignored by this proposal.  I just don't want to see further seperation
> between user bases.

I beat that drum for years.  I've become convinced that it's a lost cause.

---

From: Henry Spencer <[email-masked]>
Date: Sun, 26 Jun 2005 13:59:17 -0400 (EDT)

On Fri, 24 Jun 2005, Steve Walker wrote:
> ...In the real world, if all clients must support mime 

There's nothing which requires a MIME-supporting client to use MIME in all
articles posted through it.

For one thing, an article which is written in plain ASCII text doesn't
need to be a MIME message.  Some clients might throw in

	MIME-Version: 1.0
	Content-Type: text/plain; charset=us-ascii

just on general principles, but this doesn't actually affect the content
or interpretation of the article.  (That Content-Type header is completely
redundant -- text/plain is the default is the default content type, and
us-ascii is the default character set for text/plain.  The MIME-Version
header merely adds a -- rather weak -- assurance that in the absence of a
Content-Type header, the article really is plain human-readable text.)

For another, just because a client supports MIME doesn't mean it can't
support other conventions as well.  MIME might not even be the default;
the MIME support might be activated only by explicit user request.  All
our drafts require is that MIME support be present. 

                                                          Henry Spencer
                                                       [email-masked]

---

From: Henry Spencer <[email-masked]>
Date: Sun, 26 Jun 2005 14:08:56 -0400 (EDT)

On Fri, 24 Jun 2005, Steve Walker wrote:
> integers are used for article counters?

Our draft says nothing about the number of bits used in an article
counter, because we don't have the concept of an article counter. 

The only place where we deal with any form of article numbering is in an
explanatory note on our discussion of the Xref header, where we note that
Xref "article locators" are commonly decimal numbers.  We say nothing
about how long those numbers might be, since we're dealing with them in
ASCII text, not in a binary representation. 

> A real-world problem facing NNTP developers...

Discuss this with the NNTP WG, please.  It's not an article-format or
article-processing issue, so it's outside our scope.

                                                          Henry Spencer
                                                       [email-masked]

---

From: Steve Walker <[email-masked]>
Date: Sun, 26 Jun 2005 15:53:33 -0500

> ASCII text, not in a binary representation. 

My apologies, I am in the wrong group for this topic.

On the other issue, has anyone proposed using headers similar to
'Content-type: raw/subtype' or 'NNTP-Format: raw/subtype'.  Which 
would mean that \r\n\0 are escaped and the line length limits 
were kept, but other than that no further restrictions on format 
or content of the article was made.

This would allow yEncoding and any future applications to used in 
a way that would not cause harm to others. Clients that didn't 
want to process articles with an unspecified format would just 
ignore them.  Specialized clients would only process articles 
with a content-type of raw looking for their own signatures.

It seems like an extremely simple solution and I believe that you 
may get the client developers to add one simple header if it 
means complying with a new standard.

Steve Walker.
NNTPServer.com

---

From: Russ Allbery <[email-masked]>
Date: Sun, 26 Jun 2005 13:52:29 -0700

Steve Walker <[email-masked]> writes:

> 'Content-type: raw/subtype' or 'NNTP-Format: raw/subtype'.  Which would
> mean that \r\n\0 are escaped and the line length limits were kept, but
> other than that no further restrictions on format or content of the
> article was made.

This would be a Content-Transfer-Encoding (CTE), not a Content-Type, and
yes, people have.  I tried to work with the author of yEnc on such a
proposal, but unfortunately didn't get anywhere.  There are reasons why
the IETF is reluctant to bless any additional CTEs, but I think that this
one could have a future and could make it through the process.  In the
absence of any interest or cooperation from the motivating software
package, however, I gave up.

---

From: Ned Freed <[email-masked]>
Date: Sun, 26 Jun 2005 13:52:12 -0700 (PDT)

> 'Content-type: raw/subtype' or 'NNTP-Format: raw/subtype'.

The use of media types to identify encodings is specifically prohibited. This
is a transfer encoding, not a media type.

> Which

Bruce already pointed out that not only is it perfectly permissible to define
additionaal content encodings for this sort of thing, at least two drafts have
been written that propose doing exactly this. Both have been allowed to
exprire. I cannot speak for Dan's draft but I let mine expire due to there
being little if any apparent interest in pursuing it from the community.

				Ned

---

From: [email-masked] (Kai Henningsen)
Date: 26 Jun 2005 16:42:00 +0200

> standard.  Why don't we write a "standard" that follows standard

As far as I'm concerned - and I expect I'm not representing any kind of  
majority here - binary traffic is a completely uninteresting part of  
Usenet. In fact, my position may be better characterized by saying that  
binary traffic isn't part of Usenet, at least in general.

In any case, software authors and users of that are certainly on record as  
not caring for IETF standards. See, for one of many examples, the yEnc  
discussions.

My only interest is in text Usenet.

MfG Kai

---

From: [email-masked] (Kai Henningsen)
Date: 26 Jun 2005 16:46:00 +0200

> issue.  Some clients are going to roll at 2^31 and some at 2^32.

Wrong working group. NNTPEXT is <--- thataway. (Incidentally, the update  
to the base standard just passed IESG review, so you're a little late for  
any changes; on the other hand, towards the end, the group was nearly  
exclusively manned by server software authors ...)

MfG Kai

---

From: Arnt Gulbrandsen <[email-masked]>
Date: Sun, 26 Jun 2005 23:56:23 +0200

IMNSHO, a protocol extension will succeed only if implementing it brings 
value to the user. If implementing e.g the SMTP MUMBLE extension 
doesn't benefit either the implementor or the user, and does benefit 
someone else greatly, then that extension isn't going to be implemented 
very much.

I may be wrong, of course. But let's suppose I'm right. In that case:

1. Mobile MUAs get considerable benefit from compressing messages as 
they go from/to the server, so an SMTP/POP/IMAP extension to provide 
compression for all messages or all attachments ought to be fairly 
widely implemented by authors of such MUAs (and likely seldom by 
others).

2. Even mobile MUAs don't get much benefit from c-t-e: deflate, since 
that works only when both the message sender and all receivers support 
it, which'll be fairly seldom. Authors of clients who assume ethernet 
to the server almost never get any benefit.

Arnt

---

From: Steve Walker <[email-masked]>
Date: Sun, 26 Jun 2005 19:17:57 -0500

> is a transfer encoding, not a media type.

I was not referring to an official MIME type at all.  I was 
suggesting a new header that signifies the following body is not 
MIME.  If Content-type is too much like MIME then use something 
else like NNTP-Format.  Some form of meta data is needed to 
officially separate the MIME articles from the 5 million non-MIME 
articles that arrive every day.

If you push a standard that says everything must be mime and 85% 
of articles are non-mime there will be problems.  If people 
expect non-mime articles they will write better software.

Steve Walker
NNTPServer.com

---

From: Steve Walker <[email-masked]>
Date: Sun, 26 Jun 2005 19:30:36 -0500

> binary traffic isn't part of Usenet, at least in general.

I can understand someone not liking the expenses involved in 
running a news server due to the traffic caused by binaries.  I 
can also understand someone not liking what someone posts, be it 
binary or text.  But I can't imagine how anyone could say that a 
binary article is not Usenet.  If one person shares a written 
story and another shares an audio book of the same story, what 
makes one Usenet and the other not?  Is it the size of the 
article or the file type?  Or is it the person?

Steve Walker
NNTPServer.com

---

From: Seth Breidbart <[email-masked]>
Date: Sun, 26 Jun 2005 21:28:23 -0400 (EDT)

> on the suggestion of requiring article numbers to be 64 bit.

OK, I'll comment on it.  What's an "article number"?  Is that some
local thing that different sites do differently and nobody non-local
needs to care about?

Seth

---

From: Frank Ellermann <[email-masked]>
Date: Mon, 27 Jun 2005 04:40:05 +0200

Steve Walker wrote:

> Some form of meta data is needed to officially separate the
> MIME articles from the 5 million non-MIME articles that
> arrive every day.

How about this:

 MIME-Version: 1.0
 Content-Type: text/x-yenc; charset=unknown-8bit (RfC 1428)
 Content-Transfer-Encoding: 8bit

If I understand RfC 2046 correctly that should have a similar
effect like application/octet-stream for any newsreader not
knowing what to do with it, see "4.1.4 Unrecognized Subtypes".

Or why not this:

 MIME-Version: 1.0
 Content-Type: application/x-text-with-yenc
 Content-Transfer-Encoding: 8bit

> If you push a standard that says everything must be mime
> and 85% of articles are non-mime there will be problems.

If adding one of the above header-blocks is asking too much
from these UAs, then they'll be treated as "non-conformant"
violating the first MUST in usefor-04 section 2.3

It also says "MUST support 2231" - I'm 100% sure that my UA
doesn't support 2231.  Or to say it more clearly, I very much
doubt that even ONE user-agent supporting all 2231-oddities
in a MIME boundary parameter exists.

Now that asks for a new ticket, the minimal MIME conformance is
defined in 2049, and a "MUST support 2231" rendering precisely
ALL (= my guess) UAs as non-conformant is obviously wrong.  At
best it can be a SHOULD.
                        Bye, Frank

---

From: Russ Allbery <[email-masked]>
Date: Sun, 26 Jun 2005 19:59:38 -0700

> knowing what to do with it, see "4.1.4 Unrecognized Subtypes".

Actually, I believe that:

    MIME-Version: 1.0
    Content-Type: application/x-yenc
    Content-Transfer-Encoding: x-yenc

does pretty much what you would want from the perspective of a MIME client
that doesn't understand yEnc.  The unrecognized CTE pretty much tells
everything "hands off" in a particularly thorough manner.  RFC 2045:

   Implementors may, if necessary, define private Content-Transfer-
   Encoding values, but must use an x-token, which is a name prefixed by
   "X-", to indicate its non-standard status, e.g., "Content-Transfer-
   Encoding: x-my-new-encoding". [...]

   Unlike media types and subtypes, the creation of new Content-
   Transfer-Encoding values is STRONGLY discouraged, as it seems likely
   to hinder interoperability with little potential benefit

[...]

   Any entity with an unrecognized Content-Transfer-Encoding must be
   treated as if it has a Content-Type of "application/octet-stream",
   regardless of what the Content-Type header field actually says.

The only thing it doesn't do is help a MIME client actually understand the
body as opposed to realize that it can't deal with it.

Structurally, yEnc is actually a multipart, but since it doesn't use MIME
bounaries, one can't tag it that way.  (Not to mention that multiparts
aren't allowed to use other CTEs.)

Anyway, you can't use CTE 8bit for yEnc data.  RFC 2045 says:

   "8bit data" refers to data that is all represented as relatively
   short lines with 998 octets or less between CRLF line separation
   sequences [RFC-821]), but octets with decimal values greater than 127
   may be used.  As with "7bit data" CR and LF octets only occur as part
   of CRLF line separation sequences and no NULs are allowed.

and yEnc breaks the line length requirement.  The only already-defined CTE
you can use for yEnc data is binary, which of course NNTP and most news
software can't handle since it requires nul-cleanliness and places no
special meaning on CR and LF.

---

From: Frank Ellermann <[email-masked]>
Date: Mon, 27 Jun 2005 05:12:22 +0200

Henry Spencer wrote:
 
> There's nothing which requires a MIME-supporting client to
> use MIME in all articles posted through it.

There is, see 2049:

| (8) Conformant user agents are required, if they provide
|     non-standard support for non-MIME messages employing
|     character sets other than US-ASCII, to do so on
|     received messages only. Conforming user agents must not
|     send non-MIME messages containing anything other than
|     US-ASCII text.

|     In particular, the use of non-US-ASCII text in mail
|     messages without a MIME-Version field is strongly
|     discouraged as it impedes interoperability when sending
|     messages between regions with different localization
|     conventions. Conforming user agents MUST include proper
|     MIME labelling when sending anything other than plain
|     text in the US-ASCII character set.

Steve got us, both MUST in usefor-04 (2.3) are utter dubious.

> just because a client supports MIME doesn't mean it can't
> support other conventions as well.

For UUE and other variations of 7bit no problem, but for any
8bit conventions, how should it post this stuff ?  Bye, Frank

---

From: Henry Spencer <[email-masked]>
Date: Mon, 27 Jun 2005 00:00:23 -0400 (EDT)

On Mon, 27 Jun 2005, Frank Ellermann wrote:
> There is, see 2049...

The passages you note cover only articles containing non-ASCII text. 
Nothing compels MIME use for articles containing only ASCII characters
(possibly as a result of using higher-level conventions to encode
non-ASCII material in ASCII). 

> 8bit conventions, how should it post this stuff ?

Non-standardly.  Here, I was discussing software support for conventions,
not standards conformance, refuting the bizarre contention that a client
which supports MIME couldn't possibly support any alternative.  Clients
can have non-standard modes as well as standard ones; the non-standard
modes might even be the defaults.  This is arguably a Bad Thing, but it is
unquestionably possible. 

                                                          Henry Spencer
                                                       [email-masked]

---

From: Frank Ellermann <[email-masked]>
Date: Mon, 27 Jun 2005 10:50:33 +0200

> The passages you note cover only articles containing

Yes, AFAIK yenc is 8bit minus NUL, CR, LF, and optional tricks
to avoid trailing SP or HT.  All octects %x01..%xFF reserving
CRLF for line ends, and using "=" as an escape character.  So
that's certainly not 7bit or ASCII, whatever else it might be.

OTOH it's also not really binary, it's some kind of 8bit.

>> how should it post this stuff ?

> Non-standardly.  Here, I was discussing software support for
> conventions, not standards conformance

That's not allowed in usefor-0?, all UAs MUST follow RfC 2049,
and 2049 says "post either MIME or ASCII".  That's what Steve
was talking about, conformance with our standard includes this
one MUST.

> Clients can have non-standard modes as well as standard ones

Then they'd need a SHOULD instead of a MUST in usefor to _post_
such articles.  Bye, Frank

---

From: Frank Ellermann <[email-masked]>
Date: Mon, 27 Jun 2005 11:07:28 +0200

Russ Allbery wrote:

> you can't use CTE 8bit for yEnc data.  RFC 2045 says:
> "8bit data" refers to data that is all represented as
>  relatively short lines with 998 octets or less between
>  CRLF line separation sequences [RFC-821]), but octets
>  with decimal values greater than 127 may be used.  As
>  with "7bit data" CR and LF octets only occur as part

> and yEnc breaks the line length requirement.

I'm not sure, seen in file YENC_1_3.TXT (2002-07-22 on my box):

| To facilitate transmission via existing standard protocols
| (most notably NNTP), carriage return/linefeed pairs should be
| written to the output stream after every n characters, where
| n is the desired line length.  Typical values for n are 128
| and 256.

That's smaller than 998.  So it should be okay for 8bit MIME,
although anything else is more like UUE than MIME.  Bye, Frank

---

From: Bruce Lilly <[email-masked]>
Date: Mon, 27 Jun 2005 08:01:18 -0400

[...]
>  Content-Transfer-Encoding: 8bit

The problem with both is that there is no satisfactory indication
of the media type.  In the first case, you're claiming the the
unencoded content is text of some sort. In the second, you're
claiming application data.  In practice, yEnc is used for image,
video, and audio media, possibly also application data; I doubt
that is has been used for text.  The idea (granted, not 100% clear
from the specifications) is that a MIME media type identifies what
the (unencoded) content is, and Content-Transfer-Encoding
identifies two things:
1. An encoding method (possibly an "identity" encoding)
2. The characteristics ("domain" in RFC 2045) of the encoded
   data (7bit, 8bit, binary).

YEnc is clearly a transfer encoding, not a media type.

---

From: Bruce Lilly <[email-masked]>
Date: Mon, 27 Jun 2005 08:15:39 -0400

On Mon June 27 2005 04:50, Frank Ellermann wrote:

> to avoid trailing SP or HT.

The key here is "AFAIK". The problem is that "n" (from your other
message) isn't specified to be constrained to 998 or less, and in
any event is not formally specified in a stable specification
suitable as the basis for an IETF encoding.  That has led to
interoperability problems (as noted by Russ), and the proponents
of yEnc seem to be unwilling to standardize the specification in
the IETF (as noted by several people).

> OTOH it's also not really binary, it's some kind of 8bit.

iff n <= 998
 
> was talking about, conformance with our standard includes this

N.B. "conformance with our" specification (it's not a standard).
 
> such articles.  Bye, Frank

No, there are no IETF brown shirts forcing conformance at gunpoint.
No matter what we say (given the constraint that what we say has to
pass IESG muster), there will be non-conforming software.  Our task
is to specify format, protocol, and best practice for those who wish
to build conforming implementations.

---

From: Ned Freed <[email-masked]>
Date: Mon, 27 Jun 2005 07:42:20 -0700 (PDT)

> the IETF (as noted by several people).

Bingo. And in this discussion we now have a proposal to identify this stuff
with either 'Content-type: raw/subtype' or 'NNTP-Format: raw/subtype' label and
a refusal to consider using a CTE. The problems with both are obvious and don't
bear repeating.

I give up. It is clear that there's no real interest in moving forward on this
in a way that's compatible with existing standards. So, if any proposals for
additional CTEs move forward, they won't do so as part of an attempt to bridge
this gap in usenet, but rather based solely on their own merit.

				Ned

---

From: Henry Spencer <[email-masked]>
Date: Mon, 27 Jun 2005 11:08:24 -0400 (EDT)

On Mon, 27 Jun 2005, Frank Ellermann wrote:
> Yes, AFAIK yenc is 8bit minus...

Correct.  I was speaking more generally, not of the specific question of
reconciling yenc with MIME. 

> That's not allowed in usefor-0?, all UAs MUST follow RfC 2049...

How many times do I have to say "non-standard"? :-)  Here I was talking
about what is technologically possible, not what is standard-compliant. 

> such articles.

If they're in a non-standard mode, what's in Usefor is irrelevant. 
Standards cannot dictate what happens when the standard is not
followed.

                                                          Henry Spencer
                                                       [email-masked]

---

From: Steve Walker <[email-masked]>
Date: Mon, 27 Jun 2005 10:52:29 -0500

> to build conforming implementations.

The percentage, which is greater than 65% and growing, of 
non-conformance is too large to just be ignored.  It is not in 
the best interest of Usenet to write a standard that software 
client developers can not conform to even if they would like to.

Is this group trying to writing something that standardizes on 
existing practices or is it trying to re-invent a whole new 
protocol.  In that case, a proposal for Usenet-Version: 2.0 is in 
order and you would not have to keep backward compatibility.

> 2.3  MIME Conformance
> 
>    User agents MUST meet the definition of MIME-conformance in [RFC2049]
>    and MUST also support [RFC2231].  This level of MIME Conformance
>    provides support for internationalization and multimedia in message
>    bodies ([RFC2045], [RFC2046], [RFC2231]), and support for
>    internationalization of headers ([RFC2047], [RFC2231]).  Note that
>    [Errata] currently exist for [RFC2046] and [RFC2231].

Would changing the above to the following harm anything?

User agents SHOULD meet the definition of MIME-conformance in 
[RFC2049] and SHOULD also support [RFC2231].

The non-binary focused clients would support mime because it 
works well for text and the binary focused clients most likely 
would not implement a full MIME parser.  The mime clients should 
be looking for the header MIME-Version anyway and they could 
ignore articles without it.

Steve Walker
NNTPServer.com

---

From: Sebastian Brocks <[email-masked]>
Date: Mon, 27 Jun 2005 18:12:54 +0200

Steve Walker schrieb:
> The mime clients should 

No they could not ignore articles without it, those articles are just as
valid as MIME articles if they do not contain 8 Bit Characters.

---

From: Seth Breidbart <[email-masked]>
Date: Mon, 27 Jun 2005 12:19:35 -0400 (EDT)

Steve Walker <[email-masked]> wrote:

> non-conformance is too large to just be ignored.

Where are you getting that number?  Are you including plaintext
articles that have only ASCII characters and lack MIME headers?  Those
_are_ valid MIME.

> protocol.

It's trying to standardize those existing practices which can be
standardized.  Apparently yEnc can't be; at least, the efforts so far
have failed, and this is the wrong group to work on standardizing yEnc
anyway, for any number of reasons.

If you want us to point to a yEnc standard in addition to the MIME
standard, show us the one to point to.

Seth

---

From: Russ Allbery <[email-masked]>
Date: Mon, 27 Jun 2005 09:25:36 -0700

> although anything else is more like UUE than MIME.  Bye, Frank

Oh, hm, that's interesting.  I hadn't realized that was in there (maybe
it's a change from the original protocol?).  That would seem to indicate
you could get away with content-type application/x-yenc, CTE: 8bit if you
really wanted to (and as I recall, yEnc includes a variety of other data
besides the actual message, in an attempt to -- poorly -- duplicate MIME
multipart features, so it's arguably not purely a CTE either).

---

From: Frank Ellermann <[email-masked]>
Date: Mon, 27 Jun 2005 18:28:09 +0200

Henry Spencer wrote:

> How many times do I have to say "non-standard"? :-)

Until you've submitted s-o-1036 as FYI... <gd&r>

Joke off, from my POV non-standard is the gap between
SHOULD and MUST.  But violating a MUST is not only
"non-standard", it's broken and known to be harmful
in some cases.  Not all cases, that would be OBVIOUS,
and OBVIOUS is a 2119-keyword that MUST NOT be used -
consequently it's not mentioned in 2119.

Unfortunately Bruce's keywords are slightly different,
SHOULD is a "mere recommendation", MUST is important,
and violating a MUST is a real problem if the brown
shirts get you.  Otherwise it's probably "let's see
what happens".

No surprise that we have serious difficulties to come
to a common understanding of what "UAs MUST conform
to RfC 2049 and MUST support RfC 2231" actually means.

 From my POV it's perfectly sane to reject articles if
they use 8bit without MIME.  It's the idea of saying
MUST instead of SHOULD.

 From your or Bruce's POV this MUST apparently works
like a SHOULD in my terminology, if it's broken you
own the pieces, no other consequences.  This includes
many fine points like "no justification to reject or
clean such articles", "no valid reason for an abuse
report", etc.
                       Bye, Frank

---

From: Henry Spencer <[email-masked]>
Date: Mon, 27 Jun 2005 12:33:14 -0400 (EDT)

> client developers can not conform to even if they would like to.

All other issues aside, there is simply no way we can standardize the yenc
stuff without cooperation and participation from the community that uses
it... and as others have noted, in the past that community has been
conspicuously uninterested. 

This isn't "65% and growing" of Usenet; this is, for all practical
purposes, a separate network which happens to use a few of the same
protocols and some of the same servers. 

> existing practices or is it trying to re-invent a whole new 

We're trying to standardize existing practices, specifically the ones
derived from the earlier quasi-standards for news, notably RFC 1036. 
There is a limit to what we can include under that umbrella; the yenc
people have, to put it bluntly, gone their own way with complete disregard
for standards.  There is every reason to believe that they will continue
to do so, and in that context, contorting standards to accommodate them
appears rather pointless. 

> ignore articles without it.

As you've been told repeatedly, just because a client is MIME-capable
doesn't mean that it insists on handling only MIME messages.  Most
plain-text articles on Usenet today do not have MIME-Version headers.
A Usenet client which ignored any article lacking a MIME-Version header
would be useless. 

                                                          Henry Spencer
                                                       [email-masked]

---

From: Russ Allbery <[email-masked]>
Date: Mon, 27 Jun 2005 09:47:27 -0700

Henry Spencer <[email-masked]> writes:

> plain-text articles on Usenet today do not have MIME-Version headers.  A
> Usenet client which ignored any article lacking a MIME-Version header
> would be useless.

I haven't run more complete statistics, but assuming that my randomly
chosen text newsgroup is indeed a decent random sample, this is
interestingly not the case.  Around 60% of articles had a MIME-Version
header.

I'd have to do more work to see if this applies across the rest of the
text Usenet.

(The rest of your point, of course, I agree with completely.)

---

From: Henry Spencer <[email-masked]>
Date: Mon, 27 Jun 2005 12:55:42 -0400 (EDT)

On Mon, 27 Jun 2005, Russ Allbery wrote:
> > ...Most
> > plain-text articles on Usenet today do not have MIME-Version headers...
> 

I ran a quick check in a couple of groups I follow regularly, and got
numbers broadly agreeing with that.  I'm surprised; I hadn't thought it
was that high yet.

                                                          Henry Spencer
                                                       [email-masked]

---

From: Frank Ellermann <[email-masked]>
Date: Mon, 27 Jun 2005 19:12:30 +0200

Bruce Lilly wrote:

>> AFAIK yenc is 8bit minus NUL, CR, LF, and optional tricks

> The key here is "AFAIK".

You can check it at <http://www.yenc.org> - this page starts:

| 02.April.2003:
| A new encoding has been specified by Ned Fried.
| gZip-8Bit is an extension to yEnc with MIME and integrated
| compression.

| The next round in standardization of Binary Usenet messages
| has been started.

I've no idea why that's apparently the last word about yenc.

> The problem is that "n" (from your other message) isn't
> specified to be constrained to 998 or less

For the mentioned "typical values" 128 or 256 I don't see any
problem, apparently any "n" up to 997 goes (they don't allow
to split an escaped NUL, HT, CR, LF, SP, or "=" by CRLF, and
so they need "n+1" for such cases),

> not formally specified in a stable specification suitable as
> the basis for an IETF encoding

Yes, I'm guessing what it means.  OTOH this part of yenc is
simple and clear enough to get it right for NNTP.  But when
they start with =ybegin, multiple parts, and Subject tags I'm
not more interested, after all I'm not planning to implement
it.  JFTR, I also don't plan to implement message/partial.

But I have an application claiming to create or at lest read
yenc (among others like UUE and XXE).  Not message/partial -
I've never seen any message/partial, whatever that means, it
could be because I rarely look into binary groups, not once
for at least 20 months.

> the proponents of yEnc seem to be unwilling to standardize
> the specification in the IETF (as noted by several people)

<shrug />  If Steve really wants it he's free to try it, my
interest is limited to the question of our various MUSTs, and
to a certain degree to MIME itself.  In your 822-discussion
limited to the one case _without_ gzip.

> "conformance with our" specification (it's not a standard).

We're going for "proposed standard", not "informational" or
"experimental".  Maybe BCP for USEAGE.  Modulo "with the iESG
you never know".
                 Bye, Frank

---

From: Frank Ellermann <[email-masked]>
Date: Mon, 27 Jun 2005 19:26:00 +0200

Bruce Lilly wrote:
 
> The problem with both is that there is no satisfactory
> indication of the media type.

Yes, isn't that the idea of application/octet-stream ?
Some binary crap.  Actually the same case as UUE in a
plain text part, the only difference, it's not 7bit.

> YEnc is clearly a transfer encoding, not a media type.

If that's "clear" then it would be also "clear" for UUE.
The latter isn't the case, your premise is utter dubious.

Both are not designed to fit nicely into a MIME framework,
no surprise for UUE.  But the framework is still capable
to transport this crap as is without causing havoc.  Bye

---

From: Ned Freed <[email-masked]>
Date: Mon, 27 Jun 2005 11:11:23 -0700 (PDT)

> I've no idea why that's apparently the last word about yenc.

It was the last word from me because basically all I got for my trouble was a
bunch of (mostly private) hate mail and precious little public (or private)
support for the idea.

				Ned

---

From: Ned Freed <[email-masked]>
Date: Mon, 27 Jun 2005 11:23:22 -0700 (PDT)

> If that's "clear" then it would be also "clear" for UUE.

Actually, it is clear that uuencode is a CTE. It just isn't a very good one -
there are numerous variations, some compatible, others not. Issues include line
length, the use or non-use of spaces versus tildes, how incomplete lines are
handled, and the format ofd both the begin and end lines.

Messages with a CTE of X-UUENCODE were fairly common at one point, as a matter
of fact.

Similar issues surround the class of 4-in-5 encodings: btoa, base85, and so on.

There used to be a document floating around that described some, but not all,
of the UUENCODE variants, but I've long since lost track of it.

				Ned

---

From: Bruce Lilly <[email-masked]>
Date: Mon, 27 Jun 2005 15:06:46 -0400

> "experimental".  Maybe BCP for USEAGE.  Modulo "with the iESG

"Not all RFCs are standards".  More important "The Internet-Draft should
neither state nor imply that it has any standards status; to do so
conflicts with the role of the RFC Editor and the IESG." (1id-guidelines).
I think I mentioned this before; the draft(s) need to be checked
carefully by the Editors (this should already have been done in
accordance with the guidelines...).

Specifically, the draft contains about a dozen instances of "this standard"
(one says "this present standard"), there's a reference to "other standards"
(implying that the draft is a standard), a reference to "standard
User-Agent", etc.   The only occurrence that shouldn't be changed is the
one in the IPR boilerplate.

---

From: Bruce Lilly <[email-masked]>
Date: Mon, 27 Jun 2005 15:13:54 -0400

[...]
> problem,

The problem is that n = 999, n= 1000, ... n = 99999999999999, ...
are permitted.

---

From: Ken Murchison <[email-masked]>
Date: Mon, 27 Jun 2005 15:17:47 -0400

> bunch of (mostly private) hate mail and precious little public (or private)

Ned,

Just out of curiosity, did you ever look at Jeremy Nixon's work on this?

http://www.exit109.com/~jeremy/news/binaries/

It doesn't include compression, but it does seem to retain backwards 
compatibility.

-- 
Kenneth Murchison     Oceana Matrix Ltd.
Software Engineer     21 Princeton Place
716-662-8973 x26      Orchard Park, NY 14127
--PGP Public Key--    http://www.oceana.com/~ken/ksm.pgp

---

From: Bruce Lilly <[email-masked]>
Date: Mon, 27 Jun 2005 15:19:25 -0400

> of the UUENCODE variants, but I've long since lost track of it.

Maybe http://www.alvestrand.no/ietf/uuencode-no.html ?

---

From: Russ Allbery <[email-masked]>
Date: Mon, 27 Jun 2005 12:34:35 -0700

>> It was the last word from me because basically all I got for my trouble
>> was a bunch of (mostly private) hate mail and precious little public
>> (or private) support for the idea.

> It doesn't include compression, but it does seem to retain backwards

I don't really see a lot of point in the compression aspect of this,
actually.  At least for the Usenet problem, the data is already
compressed, and converting to base64 and then compressing seems like a lot
of hassle.

The real point of yEnc is that it specifies the minimum possible munging
of binary data to allow it to be transferred through Usenet.  (And
probably most mail servers, but that I haven't investigated at all.)  As a
result, it saves a lot of encoding space without requiring the harder work
of switching to a full binary transport.

One of the reasons why standardizing that particular good idea has been so
hard is that the author of that very solid idea then went on to add a
bunch of uuencode-inspired cruft that reinvents MIME multiparts poorly
(albeit with a few good ideas that could usefully be added to MIME
multiparts), and he was insisting on dragging along all of that cruft in
any standardization effort.

---

From: Frank Ellermann <[email-masked]>
Date: Mon, 27 Jun 2005 22:12:59 +0200

Bruce Lilly wrote:

>> UUENCODE variants, but I've long since lost track of it.
 
> Maybe http://www.alvestrand.no/ietf/uuencode-no.html ?

Nice, thanks, that zoo was fortunately replaced by B64 and
MIME.  For "yenc" it's the same zoo again, trying to solve
several problems.  Ned wrote:

>> Actually, it is clear that uuencode is a CTE.

The main part of it, binary IN, some text OUT, could be a
CTE.  But it also had some kind of a Content-Disposition
(specifying filename), IIRC it also allowed the indication
of "executable".

I didn't read the old CTE-drafts, did one of them cover the
CTE-part of "yenc" for some value of n <= 997 ?  Bye, Frank

---

From: Frank Ellermann <[email-masked]>
Date: Mon, 27 Jun 2005 22:33:33 +0200

Bruce Lilly wrote:
 
> the draft contains about a dozen instances of "this standard"

Okay, that's wrong, minus a case introduced by the boilerplate.

But we're still going for "proposed standard", and saying "our
standard" in a WG discussion is perfectly okay, because we've
to make sure that all works as soon as it is a standard.

Declaring all pre-2231 UAs as "non-conformant" is an odd idea,
normally that's a case for a SHOULD meaning "old UAs have an
excuse, but new UAs need another _good_ excuse if they don't
support 2231".  So why a MUST in this case, jumping from 1036
to MUST 2231 is wrong.  Jumping from s-o-1036 to MUST 2049 is
okay from my POV, but Steve says that even this is too much :-(
 
                      Bye, Frank

---

From: Ned Freed <[email-masked]>
Date: Mon, 27 Jun 2005 14:03:30 -0700 (PDT)

> > Maybe http://www.alvestrand.no/ietf/uuencode-no.html ?

I actually think someone took this and generalized it at some point, but I
cannot seem to track down the reference in my email archives and google isn't
being helpful today.

> (specifying filename), IIRC it also allowed the indication

Uuencode typically allows the optional specification of a file name and an
octal protection mask. What's actually allowed as a file name varies, as does
what's allowed in the protection field. There are variants that also include a
checksum (on the end line), typically CRC16 based.

But none of this makes UUENCODE ineligable as a CTE. If it did base64 would
also have been ineligable, as the original specification of it allowed for
embeeded comments (and maybe some variably in terminator representation - I no
longer recall). In the case of base64 we simply dropped the comment facility,
had we gone with base64 we would have dealt with the filename, protection, and
other gunk by calling for it to be assigned a fixed value on encode and ignored
on decode.

We elected to go with base64 and not with uuencode for two main reasons: (1)
Uuencode's character repetoire exceeded the minimal safe subset of ASCII, and
at the time there were a fair number of systems that used national variants of
ASCII and had trouble with characters that uuencode uses and (2) All the format
variations were already causing interop problems, and even if we nailed things
down in our specificaiton the temptation to use just existing software would
surely have led to more problems.

Similar issues apply to the 4-in-5 encodings. In particular, there just aren't
enough safe characters in the minimal safe subset of ASCII to do base85. And
there are plenty of format variations, including some really wild ones, in the
4-in-5 space.

In retrospect I kind of wish we'd ignored the minimal safe subset issue and
gone with a 4-in-5 (preferably the one that's built into PostScript since it is
fully and completely specified), since 20% overhead is a lot more attactive
than 33%. But you have to remember that the entire MIME effort started not as
an attempt to send non-text material through email. Rather, the main goal was
to internationalize text email. It is unlikely in the extreme that given the
original purpose of the effort we would have allowed to ignore what was then a
fairly critical factor in the handling of internationalized text.

> CTE-part of "yenc" for some value of n <= 997 ?  Bye, Frank

Yes, that's exactly what my draft does (or did).

				Ned

---

From: Frank Ellermann <[email-masked]>
Date: Tue, 28 Jun 2005 00:42:16 +0200

Ned Freed wrote:

> an octal protection mask

Yes, now I recall it, user group world, odd was "dangerous".
When my OS/2 box dies I'll proably meet these bits again.

> a checksum (on the end line), typically CRC16 based.

More dark and bad memories, and now they tell us that MD5
is obsolete.

> at the time there were a fair number of systems that used
> national variants of ASCII

Yes, the German variant was very ugly.  I also didn't like
the trigraph stuff in C.

> plenty of format variations, including some really wild
> ones, in the 4-in-5 space.

One I vaguely recall is "FIFfy" (four in five) used for 
binary file transfers in German videotext (like Prestel).

>> I didn't read the old CTE-drafts, did one of them cover
>> the CTE-part of "yenc" for some value of n <= 997 ?
 
> Yes, that's exactly what my draft does (or did).

Then getting an RfC for it is a good idea.  I'm only lost
why it should integrate gzip, compression algorithms come
and go, they never satisfy all users or all applications.

---

From: Henry Spencer <[email-masked]>
Date: Mon, 27 Jun 2005 20:03:34 -0400 (EDT)

> More dark and bad memories, and now they tell us that MD5

MD5 is still an excellent choice when the concern is accidental
corruption, rather than deliberate human malice.

                                                          Henry Spencer
                                                       [email-masked]

---

From: "Charles Lindsey" <[email-masked]>
Date: Mon, 27 Jun 2005 17:09:24 GMT

>everything "hands off" in a particularly thorough manner. ...

Yes, the Yenc people have been told, repeatedly, that they could start using
    Content-Transfer-Encoding: x-yenc
tomorrow without breaking any rules. But they just don't want to listen.
What is abundantly clear is nothing which requires special keywords in
the Subject header for it to work is _ever_ going to get to be an IETF
standard.

Before reading Russ's message, I was about to propose:

    Content-Type: application/octet-stream
    Content-Transfer-Encoding: 8bit

as a way of legitimizing current Yenc practice (see remark from Frank, I
believe, that Yenc encourages lines to be kept well within 998).

-- 
Charles H. Lindsey ---------At Home, doing my own thing------------------------
Tel: +44 161 436 6131 Fax: +44 161 436 6133   Web: http://www.cs.man.ac.uk/~chl
Email: [email-masked]      Snail: 5 Clerewood Ave, CHEADLE, SK8 3JU, U.K.
PGP: 2C15F1A9      Fingerprint: 73 6D C2 51 93 A0 01 E7 65 E8 64 7E 14 A4 AB A5

---

From: Russ Allbery <[email-masked]>
Date: Mon, 27 Jun 2005 19:25:09 -0700

> believe, that Yenc encourages lines to be kept well within 998).

Bruce's message is entirely correct; they should really be using:

    MIME-Version: 1.0
    Content-Type: image/jpeg

or whatever the actual content of the thing is so that software knows what
this blob of data is supposed to decode to.

All the additional metadata and multipart stuff is still a problem,
though.  If it's actually a multipart message, I don't think there's any
way to get around needing application/x-yenc right now (and of course,
yEnc doesn't have any way of embedding the file type because the author of
yEnc didn't understand the need, so you have to play extension guessing
games to figure it out after that, with all the problems that go with
that).

---

From: Steve Walker <[email-masked]>
Date: Mon, 27 Jun 2005 21:33:26 -0500

> articles that have only ASCII characters and lack MIME headers?  Those

I am counting only yEncoded articles in that number and only the 
ones that bother to include the word yEnc in the Subject header. 
  The number of non-conforming articles is higher than 65%.  I 
just choose to use yEnc'ed because it's the most popular article 
format, conforming or not.

> anyway, for any number of reasons.

The existing practice is posting non-mime articles.  Today the 
bulk may be yEncoded and next year it may be something else. 
It's certain that Usenet will adapt and grow.  Forcing absolute 
restrictions on the article format is a bad idea.

> standard, show us the one to point to.

Even if there was a fixed yEncoding standard I would still not 
suggest that it be used.  The article format need to remain 
flexible so that Usenet can continue to grow and adapt.   There 
needs to be someway to allow posting content without necessary 
restrictions on the exact message structure.

Steve Walker
NNTPServer.com

---

From: Russ Allbery <[email-masked]>
Date: Mon, 27 Jun 2005 19:38:52 -0700

Steve Walker <[email-masked]> writes:

> The existing practice is posting non-mime articles.  Today the bulk may
> be yEncoded and next year it may be something else. It's certain that
> Usenet will adapt and grow.  Forcing absolute restrictions on the
> article format is a bad idea.

Steve, this is the sort of statement that causes people involved in
message format standardization to tear their hair out.  This is exactly
the *point* of MIME -- it's an *extensible* framework for allowing the
message format to change and evolve.  This is the whole reason why we're
*using* it.  MIME takes care of many of the corner cases that make
changing the article format hard.

This sort of massive missing of the point is a lot of the reason why I
have a hard time caring about the binary Usenet any more.  It's exactly
what made the original yEnc discussions so frustrating.  I and others put
a lot of time into trying to get yEnc to be deployable in the right way,
and was basically told that our input was not of interest, that
standardization was a waste of time, and that they found efforts to make
things interoperate properly offensive.

Not responsible for advice not taken.  We can't standardize the work of
people who are actively opposed to standardization.  As far as I'm
concerned, enough of that reaction and one forfeits one's seat at the
table.  If the yEnc folks are happy to be unstandardized, they can go
right ahead and continue to be unstandardized after this working group
finishes its work, as far as I'm concerned.

There is *way* too much NIH syndrome and inferior reinventions of the
wheel on Usenet historically.

In the meantime, I have problems that I want to solve, and I'm not going
to stop trying to solve them just because other people are uninterested in
cooperation.  Nor do I think that we should bend over backwards to
accomodate the needs of people who have chosen to stay actively uninvolved
in the process.

---

From: Steve Walker <[email-masked]>
Date: Mon, 27 Jun 2005 22:07:16 -0500

> A Usenet client which ignored any article lacking a MIME-Version header

If your intent is to allow millions of non-mime articles to flow 
freely each day through the Usenet why would you require all 
Usenet clients to support MIME.  The support may even be a two 
edged sword in that the client who supported mime wouldn't be 
able to post non-mime and still call themselves a compliant news 
client.

Why would software developers bother supporting any requirements, 
like line length and 7bit headers, it they are still going to be 
called non-compliant.  Having hundreds of clients each making 
their own rules would be bad for Usenet in general.

 From your charter (website):
> The Goal of this working group is to publish a standards-track
> successor to RFC 1036 that with particular attention to
>  backward compatibility

How is requiring mime backwards compatible with out providing 
some type of out (meaning SHOULD instead of MUST) for the 
existing articles that are not, and can not be, mime compliant?

RFC 1036 does not even include the word MIME.

Steve Walker
NNTPServer.com

---

From: Frank Ellermann <[email-masked]>
Date: Tue, 28 Jun 2005 05:13:21 +0200

> RFC 1036 does not even include the word MIME.

That's no surprise, 1036 < 1341 < 1521 < 2045.

OTOH I get 79 occurences of MIME in 78 lines
of s-o-1036.  Only 26 in 24 lines of usefor-04.

              Bye, Frank

---

From: Henry Spencer <[email-masked]>
Date: Mon, 27 Jun 2005 23:17:30 -0400 (EDT)

On Mon, 27 Jun 2005, Steve Walker wrote:
> The existing practice is posting non-mime articles.

Depends very much on where you look.  Per earlier comments by Russ, I ran
some numbers on a sample of just under 54,000 articles, from most of the
*text* newsgroups in the traditional Usenet hierarchies, and found 58% had
a MIME-Version header.  Existing practice in the text groups *is*
increasingly to use MIME. 

See earlier comments about the binary groups being basically a separate
community. 

> Today the 
> It's certain that Usenet will adapt and grow...

Exactly.  And precisely because we can't predict just what formats will be
in use in the future, there is every reason to want some kind of standard
labeling scheme, so you can *tell* what format an article is in and
whether you know how to decode it, rather than having to guess based on
heuristics.  The labeling system needs to be extensible, so it covers data
formats we don't know about today.  And it should be non-intrusive, so it
doesn't inject garbage into the Subject header or other headers which
already have other purposes.

And that's what MIME is.  It's a little more complicated than it needs to
be, partly because there were some difficult backward-compatibility
constraints.  But by and large, MIME is a well-designed framework to
support future adaptation and growth, and make it less painful.

The reason why yenc isn't compatible with MIME is that the yenc people
don't care about anything but their own little world.  *They* are not
interested in providing for future adaptation and growth. 

That is their problem, not ours. 

                                                          Henry Spencer
                                                       [email-masked]

---

From: Steve Walker <[email-masked]>
Date: Mon, 27 Jun 2005 22:35:37 -0500

> changing the article format hard.

I can certainly see this point and I'm willing to bet that the 
yEncoder author would also see the value of MIME if they took the 
time.  It doesn't change the fact that we have a huge amount of 
non-mime articles to deal with today and for the foreseeable 
future.  We just need a little black box to shove all the 
non-mime articles into until everyone either uses mime or it's 
replaced with something better.

I think most people are agreeing that a mime compliant yEncoder 
standard isn't possible in the near future.  If you want mime and 
you don't want to break existing practice, the only choice I see 
is to not require mime.  It's the only way not to break stuff.

Steve Walker
NNTPServer.com

---

From: Russ Allbery <[email-masked]>
Date: Mon, 27 Jun 2005 20:34:22 -0700

Steve Walker <[email-masked]> writes:

> I think most people are agreeing that a mime compliant yEncoder standard
> isn't possible in the near future.  If you want mime and you don't want
> to break existing practice, the only choice I see is to not require
> mime.  It's the only way not to break stuff.

Well, I vote for requiring MIME and letting non-MIME articles just be
non-compliant.  It's not like IETF standards are some sort of government
law, or that software that doesn't comply with them will magically stop
functioning.

We need to make sure that nothing in the standard requires software to
refuse to deal with non-compliant articles and then leave the degree to
which people want to deal with them up to the individual software authors,
exactly the way that yEnc is now.

---

From: Steve Walker <[email-masked]>
Date: Mon, 27 Jun 2005 22:44:20 -0500

> That is their problem, not ours. 

It becomes everyone's problem.  What are people going to do when 
some idiot starts sending 6 million cancel messages per day.  I 
have a network/servers that can handle this load, but the smaller 
sites may not.  If Usenet doubles every 2 years in ten years you 
are looking at >100 million non-compliant article per day.

Steve Walker
NNTPServer.com

---

From: Russ Allbery <[email-masked]>
Date: Mon, 27 Jun 2005 20:45:50 -0700

>> interested in providing for future adaptation and growth. That is their
>> problem, not ours.

> It becomes everyone's problem.  What are people going to do when some
> idiot starts sending 6 million cancel messages per day.

You start filtering out their messages until their feed gets terminated.
What does that have to do with yEnc, or for that matter, with
standardization in general?

> I have a network/servers that can handle this load, but the smaller
> sites may not.  If Usenet doubles every 2 years in ten years you are
> looking at >100 million non-compliant article per day.

None of which are carried on the text servers anyway.  Besides, what
difference does that make?  What's the point of an argument from volume?
Probably 90% of HTML web pages don't comply with the HTML standard either;
does that mean that the HTML standard is a waste of time or that all those
pages should be declared compliant?  Of course not.

---

From: Seth Breidbart <[email-masked]>
Date: Tue, 28 Jun 2005 00:05:18 -0400 (EDT)

> some idiot starts sending 6 million cancel messages per day.

Yawn, ignore them and him, and depeer him.

Just like the last time.

Seth

---

From: "Charles Lindsey" <[email-masked]>
Date: Tue, 28 Jun 2005 18:08:27 GMT

>one in the IPR boilerplate.

May I point out that the MESSFOR drafts that preceded RFC 2822 also used
the words "this standard" in the same manner. Also that out earlier
arictle-* drafts contained the words:

[The use of the words "this standard" within this document when
referring to itself does not imply that this draft yet has pretensions
to be a standard, but rather indicates what will become the case if and
when it is accepted as an RFC with the status of a proposed or draft
standard.]

Those words are still in USEPRO, and ought to be in USEFOR as well, for
the removal of all doubt.

---

From: Harald Tveit Alvestrand <[email-masked]>
Date: Wed, 29 Jun 2005 10:12:57 +0200

Standard practice is to write the document the way you want the result to 
end up. That includes words like "this standard" (although, in many cases, 
"this document" is better for multiple reasons - for instance, because a 
standard may consist of more than one document).

I think it's also good practice to have a "Status of this memo (to be 
deleted by the RFC Editor)" section that explains that this is a draft that 
is intended for the standards track, and is currently a work item of the 
USEFOR WG, and that comments should be directed there. State the obvious...

--On 27. juni 2005 15:06 -0400 Bruce Lilly <[email-masked]> wrote:

> Specifically, the draft contains about a dozen instances of "this
> standard" (one says "this present standard"), there's a reference to
> "other standards" (implying that the draft is a standard), a reference to
> "standard User-Agent", etc.   The only occurrence that shouldn't be
> changed is the one in the IPR boilerplate.
>
>
>