Subject: Lines count in USEFOR

From: Julien ÉLIE <[email-masked]>
Date: Sat, 14 Nov 2009 00:34:40 +0100

Hi,

According to USEFOR:

3.3.1.  Lines

   The Lines header field indicates the number of lines in the <body>
   (as defined in [RFC5322]) of the article.

   lines           =  "Lines:" SP *WSP 1*DIGIT *WSP CRLF

   The line count is the number of CRLF separators in the <body>.

   Historically, this header field was used by the NNTP [RFC3977]
   overview facility, but its use for this purpose is now deprecated.
   As a result, this header field is to be regarded as obsolescent, and
   it is likely to be removed entirely in a future version of this
   standard.  All agents SHOULD ignore it and SHOULD NOT generate it.

If I read RFC 5322:

    body   =   (*(*998text CRLF) *998text) / obs-body

Isn't the final *998text counting for a line?
I believe the line count is the number of CRLF separators in the <body>
PLUS ONE.
(Unless I'm mistaken by the definition of the body?)

The second part of the paragraph seems unclear.  I am under the
impression that RFC 3977 says we can use Lines: in the overview
but USEFOR -- posterior to RFC 3977 -- deprecates that use.
This header field is not used by OVER in RFC 3977.

Maybe the reference to RFC 3977 should just be removed in the
paragraph.  Thus leaving NNTP in general.

-- 
Julien ÉLIE

« A good wife always forgives her husband when she's wrong. » (Rodney Dangerfield)

---

From: Russ Allbery <[email-masked]>
Date: Fri, 13 Nov 2009 15:44:36 -0800

> (Unless I'm mistaken by the definition of the body?)

NNTP as presently written cannot transfer an article that does not end in
a CRLF.  In practice, clients will just add a CRLF to any article that
doesn't already end in one, since otherwise they can't POST.

By the time we introduce some method of transfer that can handle arbitrary
data and therefore doesn't have this problem, I suspect the Lines header
will be completely obsolete.

> impression that RFC 3977 says we can use Lines: in the overview but
> USEFOR -- posterior to RFC 3977 -- deprecates that use.  This header
> field is not used by OVER in RFC 3977.

RFC 3977 says that you can *call* the :lines metadata element Lines:, but
it's still the metadata element.  Per RFC 3977, the Lines header is no
longer used to provide the basic overview information.  I think that's
what's referred to here.

-- 
Russ Allbery ([email-masked])             <http://www.eyrie.org/~eagle/>

---

From: Julien ÉLIE <[email-masked]>
Date: Sat, 14 Nov 2009 01:37:10 +0100

Hi Russ,

> doesn't already end in one, since otherwise they can't POST.

Ah yes, that final CRLF is not added by me but by my posting agent.

Then USEFOR uses a subset of the e-mail body.  And the definition of
the Lines: header is right for that subset but wrong in the general case
(which may be possible in an NNTP extension).

> longer used to provide the basic overview information.  I think that's

The impression I said I had was when I read "Historically, this header
field was used by the NNTP [RFC3977] overview facility, but its use for
this purpose is now deprecated.".
It looked to me, when reading that sentence, that this header could still
be stored in the overview defined by RFC 3977.  Which is not the case
in OVER answers.
As you explained, the sentence hints at LIST OVERVIEW.FMT.

Yet, something seems to be wrong:  "this header field"?  Shouldn't it
be "this header name"?  Because the value is never used.  (Otherwise,
its use for this purpose cannot be "deprecated" because we can no longer
retrieve the value -- it would be an unsupported feature, which is
different than deprecated.)

-- 
Julien ÉLIE

« There's a way of transferring funds that is even faster
  than electronic banking.  It's called marriage. » (Sam Kinison)

---

From: Russ Allbery <[email-masked]>
Date: Fri, 13 Nov 2009 16:45:15 -0800

> (which may be possible in an NNTP extension).

Yeah, I'd say that's correct.

In practice, the value of Lines varies so much (it's common for user
agents to set it prior to adding the signature file, for instance) that
being off by a line is minor compared to other problems.

> It looked to me, when reading that sentence, that this header could
> still be stored in the overview defined by RFC 3977.  Which is not the
> case in OVER answers.

> be "this header name"?  Because the value is never used.

The value *used* to be used.  Old versions of INN, for instance, would
copy the Lines header into the overview rather than doing their own
count and would only do their own count if there were no existing Lines
header.  That's the behavior that USEFOR is saying is deprecated.  It's
actually more than that; RFC 3977 outright forbids it.

---

From: "Charles Lindsey" <[email-masked]>
Date: Mon, 16 Nov 2009 12:45:46 GMT

>doesn't already end in one, since otherwise they can't POST.

Then I suspect USEFOR ought to have said that the CRLF at the end was
REQUIRED. Do we really want to fix that now? And what if you send it by
UUCP?

The definition in Son-of-1036 is

   The line count includes all body lines, including the signature if
   any, including empty lines (if any) at beginning or end of the body.
   (The single empty separator line between the headers and the body is
   not part of the body.)  The "body" here is the body as found in the
   posted article, AFTER all transformations such as MIME encodings.

which doew not really answer the question. But then a final <eol> was
REQUIRED by Son-of-1036.

Looking at the implementation of Cnews, if you contrive to inject a body
with no CRLF (which most user agents would not permit), the injection
process will add a CRLF.

But I suspect that I have also hacked a bug into my cnews, because if you
check the Lines header on this message, I have a suspicion you will find
it has counted my signature twice :-( .

-- 
Charles H. Lindsey ---------At Home, doing my own thing------------------------
Tel: +44 161 436 6131            Web: http://www.cs.man.ac.uk/~chl
Email: [email-masked]      Snail: 5 Clerewood Ave, CHEADLE, SK8 3JU, U.K.
PGP: 2C15F1A9      Fingerprint: 73 6D C2 51 93 A0 01 E7 65 E8 64 7E 14 A4 AB A5

---

From: Julien ÉLIE <[email-masked]>
Date: Mon, 16 Nov 2009 19:15:25 +0100

Hi Charles,

> REQUIRED.

It is REQUIRED if you use NNTP commands defined in RFC 3977 and 4644
(for streaming).  One cannot post or transmit an article without a final
CRLF; one sees a final CRLF when it retrieves the body.

But if NNTP extensions defines others commands (like the possibility to
send a gzipped article, and to also retrieve it gzipped, it may not be
necessary to have a final CRLF -- though if transmitted or retrieved
with usual NNTP commands, a final CRLF will need to be added...).

> And what if you send it by UUCP?

I have just tested a batch processed by rnews (indicating the size in bytes
of the article to process):

a) #! rnews 2129
b) #! rnews 2127

The result is that a final CRLF is always *added by the posting agent*.

a) ARTICLE <mid>
end of article
.

b) ARTICLE <mid2>
end of artic
.

-- 
Julien ÉLIE

« Nemo auditur turpitudinem allegans. »

---

From: Harald Alvestrand <[email-masked]>
Date: Tue, 17 Nov 2009 11:35:49 +0100

My memory says that we had an extensive discussion before we landed on 
the idea that the line count == the number of CRLFs in the Body.

1) It's unambiguously correct for bodies that end with CRLF and with no 
bare CRLF
2) It's clear how to count for bodies that contain bare CR or bare LF
3) It's compatible with at least some existing code for bodies that 
don't end in CRLF

I suggest we leave this one alone.

---

From: "Charles Lindsey" <[email-masked]>
Date: Wed, 18 Nov 2009 11:37:23 GMT

>I suggest we leave this one alone.

Agreed. I have no huge urge to change it. The only possible change wouold
be to decree that, contrary to RFC 5322, the final line of the <body> MUST
be terminated with a CRLF. That indeed seems to be the universal current
practice.

Interestingly, the UNIX wc command returns does not regard a final
unterminated line as countable (and Cnews at least uses wc).