Subject: Extending news to EAI

From: "Charles Lindsey" <[email-masked]>
Date: Mon, 1 Feb 2010 18:06:57 GMT

There is now an experimental protocol for UTF-8 headers in Email (RFC5335
and its relations). This was the product of the IMA WG. There has been
recent discussion of applying this to Netnews, and the conclusion seems to
be that the IMA WG is not the place to do this, and that a private draft
would be the way to do this. However, this list would be a reasonable
place to discuss it.

Essentially, under this protocol, UTF-8 may be freely used in Email
headers, but a downgrading mechanism is needed whenever mail passes to a
server that does not advertise the UTF8SMTP capability.

This is much what the USEFOR WG wanted to do in its earlier days, but the
decision was then taken to postpone it until the base documents were
complete, and then to bring it up again as an Experimental protocol. So
maybe now is the time to embark on it.

It is much easier with Netnews than with Email, since the underlying
transport (whether NNTP or UUCP) is already 8-bit clean. I would not
expect it to become the norm on the Big-8 groups for quite some time, but
it would be very useful for National hierarchies, such as the Scandinavian
ones where the inability to have Newsgroup Names with their own special
characters in them is a right pain (apparently).

So the experimental protocol would start off with the extensions allowed
by RFC5535, and then add UTF-8 in the Newsgroups header. It would be up to
individual hierarchies to encourage deployment of the experiment within
their groups.

It has already been established that the existing transport mechanisms
will move such articles around without problem. No downgrading is
envisaged except at gateways to email (at which point the mechanisms
already agreed for EAI/IMA would apply). But existing servers would cope
fairly well without modification, at least until UTF-8 newsgroup-names
were introduced.

Clearly, anyone expecting to read such articles would need a suitable
client. Some clients will already display them (Opera, for example).
otherwise, it would be up to people to install suitable user agents if
they wanted to see these articles properly (existing agents might display
such headers in a garbled form (which might be good enough in languages
which wers based on Latin alphabets). Bodies would still expect to be
covered by a Content-Type: test/plain; charset=utf-8.

With utf-8 newsgroup-names, again people who wanted to subscribe to such
groups would need suitable clients, but existing servers would serve them
once they had been persuaded to store them in their active lists. That
would require control messages that created such groups to be accepted,
and also articles submitted to moderated groups to be forwarded, so in
practice people who wanted to subscribe to such groups would need to
connect to servers which had been upgraded to cope. But the important
point is that articles would still propagate correctly through
non-upgraded servers.

Some early USEFOR drafts show how the Newsgroups header was to be
extended. In particular, it required some very strict normalization, so
that a simple byte-by-byte comparison of newsgroup-names would always
work.

Note that an experimental group dk.test.utf8-Ã¦Ã¸Ã¥ (which should show up
in UTF-8 clients properly, althoug this message is somposed in iso-8859-1)
already exists on several servers, notably on news.dotsrc.org, and this
message is crossposted there, and if a thread develops there, then well
and good. But people on that list might do better to subscribe to the
usefor mailing list (see http://www.imc.org/ietf-usefor/index.html).

-- 
Charles H. Lindsey ---------At Home, doing my own thing------------------------
Tel: +44 161 436 6131            Web: http://www.cs.man.ac.uk/~chl
Email: [email-masked]      Snail: 5 Clerewood Ave, CHEADLE, SK8 3JU, U.K.
PGP: 2C15F1A9      Fingerprint: 73 6D C2 51 93 A0 01 E7 65 E8 64 7E 14 A4 AB A5

---

From: Julien ÉLIE <[email-masked]>
Date: Wed, 3 Feb 2010 23:00:37 +0100

Hi Charles,

> by RFC5535, and then add UTF-8 in the Newsgroups header.

I think it will also require to change the possible values for "argument"
here:

   control         =  "Control:" SP *WSP control-command *WSP CRLF
   control-command =  verb *( 1*WSP argument )
   verb            =  token
   argument        =  1*( %x21-7E )

We need to be able to use UTF-8 in "argument".

Could it also be possible to extend dist-list to allow UTF-8 distributions?

   distribution    =  "Distribution:" SP dist-list CRLF
   dist-list       =  *WSP dist-name
                      *( [FWS] "," [FWS] dist-name ) *WSP
   dist-name       =  ALPHA / DIGIT
                      *( ALPHA / DIGIT / "+" / "-" / "_" )

Incidentally, may it exist hostnames in UTF-8 (for the Path: header field)
or are they always canonized in ASCII?

> will move such articles around without problem.

Is there any record of that?  Which news servers (and versions) were tested?

> that a simple byte-by-byte comparison of newsgroup-names would always

The last draft seems to be -09, Section 5.5:
    http://www.eyrie.org/~eagle/usefor/drafts/draft-ietf-usefor-article-09.txt

-- 
Julien ÉLIE

« Avez-vous remarqué qu'à table les mets que l'on vous sert vous mettent
  les mots à la bouche ? » (Raymond Devos)

---

From: "Charles Lindsey" <[email-masked]>
Date: Fri, 5 Feb 2010 14:54:14 GMT

>                      *( ALPHA / DIGIT / "+" / "-" / "_" )

Yes, there are various minor things that would need to be extended apart
from the Newgroups header (including the newsgroups line and the
checkgroups message).

>or are they always canonized in ASCII?

That is an interesting question, which would need dicussion, but does not
look like a show stopper.

>Is there any record of that?  Which news servers (and versions) were tested?

It should be documented in the early archives of this list.

Here is a typical Path for an article that reached news.dotsrc.org:

Path:
  news.sunsite.dk!dotsrc.org!news.szn.dk!news.bunk.cc!pnx.dk!
  newsfeed101.telia.com!nf02.dk.telia.net!news.tele.dk!
  feed118.news.tele.dk!not-for-mail

It is really a question of whether a server has it in its active file. If
so, then there is no reason why it should not pass straight through it.
The flooding algorithm will, as usual, find a way around any servers that
don't participate, but is will simply be a configuration affair rather
than any failure of implementation.

The real difficulty will be in user agents. I can read the danish group in
Opera, but for some reason it doesn't like yours. Ensuring there is a
working Path between your server and news.dotsrc.org might be worth a try.

>> that a si...

---

From: Julien ÉLIE <[email-masked]>
Date: Fri, 5 Feb 2010 20:32:22 +0100

Hi Charles,

> working Path between your server and news.dotsrc.org might be worth a try.

I do not have many feeds (only a dozen) and I do not see the Danish UTF-8 group
on them.  So I doubt a post to dk.test.utf8-æøå (that is to say dk.test.utf8-Ã¦Ã¸Ã¥)
on my news server will go far away...

I don't mind creating their group, though.

It is strange that Opera can read dk.test.utf8-æøå but not trigofacile.test.υτφ8...

I believe the French hierarchy fr.* may be willing to participate in the
experiment.  Creation of newsgroups are asked in fr.usenet.forums.evolution
(see "Noms de groupes en UTF-8" where one person is in favour -- nobody answered
that he was against).

-- 
Julien ÉLIE

« Un petit pås pøur møi, un grånd bønd pøur l'humanité ! » (Kerøzen)

---

From: "Charles Lindsey" <[email-masked]>
Date: Mon, 8 Feb 2010 12:18:41 GMT

>on my news server will go far away...

It is available on news.dotsrc.org, so you either need to peer with them,
or at least subscribe to them.

---

From: Julien ÉLIE <[email-masked]>
Date: Mon, 8 Feb 2010 21:17:10 +0100

Hi Charles,

> or at least subscribe to them.

I have just subscribed to news.dotsrc.org and created the dk.test.utf8-Ã¦Ã¸Ã¥
(C3 A6 C3 B8 C3 A5 in hexadecimal) newsgroup on my news server.
Amusingly, it is dk.test.utf8-ÃŠÃžÃ¥ on my ISO-8859-15 terminal (0xA6 and 0xB8
are ones of the few characters different between ISO-8859-1 and ISO-8859-15!).

A good news is that the pullnews program works fine with dk.test.utf8-Ã¦Ã¸Ã¥
so I could easily suck existing articles for them to show up on my news server
(news.trigofacile.com).

-- 
Julien ÉLIE

« Le travail n'est pas une bonne chose. Si ça l'était,
  les riches l'auraient accaparé. »