Subject: Empty description for moderated newsgroups

From: Julien ÉLIE <[email-masked]>
Date: Fri, 27 Nov 2009 22:36:04 +0100

Hi,

In USEPRO, we have:

4.2.  application/news-groupinfo

      newsgroups-line     = newsgroup-name
                               [ 1*HTAB newsgroup-description ]
                               [ *WSP moderation-flag ]
      newsgroup-description
                          = eightbit-utext *( *WSP eightbit-utext )
      moderation-flag     = SP "(" %x4D.6F.64.65.72.61.74.65.64 ")"
                               ; SPACE + case sensitive "(Moderated)"
      eightbit-utext      = utext / %d127-255

So we can have checkgroups (or newgroup control articles) with:

fr.test
fr.test.y\tA description.
fr.test.m\tA description. (Moderated)
fr.test.m.without.desc (Moderated)

I do not believe the last one is valid.  And even though USEPRO states
that "this unusual format is backward-compatible", I think the last one
will be considered as an unmoderated newsgroup whose description is
"(Moderated)".

Isn't it a problem?  Empty descriptions are explicitly allowed by the
syntax.  But when a newsgroup is moderated, I do not see well how
to handle empty descriptions in a backward-compatible way, except
if a description is mandatory in that case.

Incidentally:

3.10.3.  Original-Sender Header Field

   The content syntax makes use of syntax defined in [RFC5536]
   and [RFC5322].

         header              =/ Original-Sender-header
         Original-Sender-header
                             = "Original-Sender" ":" SP
                                  Original-Sender-content
         Original-Sender-content
                             = mailbox

But shouldn't it be, with USEFOR [RFC5536] syntax:

         fields              =/ *Original-Sender-header
         Original-Sender-header
                             = "Original-Sender" ":" SP
                                  Original-Sender-content CRLF
         Original-Sender-content
                             = mailbox

-- 
Julien ÉLIE

« -- Vous ramassez des champignons sans les connaître ?
  -- Et alors ? Ce n'est pas pour les manger mais pour les vendre. »

---

From: Harald Alvestrand <[email-masked]>
Date: Sun, 29 Nov 2009 19:44:58 +0100

> will be considered as an unmoderated newsgroup whose description is
Which software currently acts as if the last one is non-moderated?

I can see multiple ways to write a parser for this syntax; some of them 
will have this problem, other won't.
(for instance, if the check for "(Moderated)" is done independently of 
attempting to extract the description, there will not be a problem)

If it's a real problem, we should be able to cite which existing 
software has it.

---

From: Julien ÉLIE <[email-masked]>
Date: Sun, 29 Nov 2009 20:50:23 +0100

Hi Harald,

> Which software currently acts as if the last one is non-moderated?

At least INN currently does.

Example:

-=-=-=-=-=-=-=-=-=-=-=-=-=-=-=-=-=-=-=-=-=-=-=-=-=-=-=-=-=-=-=-=-=-=-=-=-=-=-
pouet.test\tTest. (Moderated)
pouet.test2 (Moderated)
-=-=-=-=-=-=-=-=-=-=-=-=-=-=-=-=-=-=-=-=-=-=-=-=-=-=-=-=-=-=-=-=-=-=-=-=-=-=-

If I feed it to controlchan (the program INN uses for checkgroups received
from peers), I receive:

-=-=-=-=-=-=-=-=-=-=-=-=-=-=-=-=-=-=-=-=-=-=-=-=-=-=-=-=-=-=-=-=-=-=-=-=-=-=-
# The following newsgroups are missing and should be added:
#
# pouet.test
# pouet.test2
#
# You can add them by executing the command(s):

/home/news/bin/ctlinnd newgroup pouet.test2 y
/home/news/bin/ctlinnd newgroup pouet.test m

# The following newsgroups descriptions are missing and should be added:
#
# pouet.test2\t(Moderated)
# pouet.test\tTest. (Moderated)
-=-=-=-=-=-=-=-=-=-=-=-=-=-=-=-=-=-=-=-=-=-=-=-=-=-=-=-=-=-=-=-=-=-=-=-=-=-=-

Note the tabulations in the descriptions found.

> I can see multiple ways to write a parser for this syntax; some of them will have this problem, other won't.
> (for instance, if the check for "(Moderated)" is done independently of attempting to extract the description, there will not be a 
> problem)

Yes, I understand.  It is possible to fix it in software.
INN just splits with:
  my ($ngname, $ngdesc) = split(/\s+/, $_, 2);
before checking for " (Moderated)".

Note that USEPRO explicitly mentions " (Moderated)" with a *leading* space.

> If it's a real problem, we should be able to cite which existing software has it.

The main problem is that RFC 3977 specifies that the format of LIST
NEWSGROUPS follows that rule:

   The newsgroups list is maintained by NNTP servers to contain the name
   of each newsgroup that is available on the server and a short
   description about the purpose of the group.  Each line of this list
   consists of two fields separated from each other by one or more space
   or TAB characters (the usual practice is a single TAB).  The first
   field is the name of the newsgroup, and the second is a short
   description of the group.

     ; newsgroups
     list-content =/ list-newsgroups-content
     list-newsgroups-content =
           *(newsgroup-name WS newsgroup-description CRLF)
     newsgroup-description = S-TEXT

     WS = 1*(SP / TAB)
     S-TEXT   = U-TEXT
     U-TEXT   = P-CHAR *U-CHAR
     P-CHAR   = A-CHAR / UTF8-non-ascii
     A-CHAR   = %x21-7E
     U-CHAR   = CTRL / TAB / SP / A-CHAR / UTF8-non-ascii

Leading spaces are only separators.  Therefore, the description
is "(Moderated)".  Not " (Moderated)".
Here, USEPRO adds a requirement to parse differently such lines.

A checkgroups was not supposed to have a different format than
the newsgroups file...

Also have a look at Son-of-1036:

    checkgroups-body    = [ invalidation ] valid-groups
                              / invalidation
    invalidation        = "!" plain-component *( "," plain-component ) eol
    valid-groups        = 1*( description-line eol )
    description-line    = newsgroup-name space description
    description         = nonblank-text [ " (Moderated)" ]

In fact, it looks as though newsgroups were required to
have a description.
It is no longer the case with USEPRO:

Note that I'm fine with that.  I just wanted to warn that
it may cause problems with existing software.
Parsing is different and, when writing the newsgroups file,
it requires to add "No description.", as it is for instance
done on <ftp://ftp.isc.org/pub/usenet/CONFIG/newsgroups>:
ab.arnet\tNo description. (Moderated)

Not easy for implementors not to walk into the trap!

As far as I am concerned, I will try to cope with that it
in the next release of INN.

-- 
Julien ÉLIE

« Audentes fortunat iuvat. » (Virgile)

---

From: Julien ÉLIE <[email-masked]>
Date: Sun, 29 Nov 2009 21:07:18 +0100

Hi Harald,

>                               [ 1*HTAB newsgroup-description ]

By the way, hopefully nobody is currently sending checkgroups with
SP instead of HTAB...

SP was allowed by Son-of-1036.

In April 2009, us.* checkgroups were using spaces.  It was not a long
time ago...
Peter J Ross noticed that:
    http://groups.google.fr/group/us.config/browse_frm/thread/7a889db82a65a9bd

(Note there were no INN bugs -- see messages #2 and confirmation in #3.)

-- 
Julien ÉLIE