Subject: [usefor] Trailing whitespace in header fields

From: Julien ÉLIE <[email-masked]>
Date: Sun, 7 Aug 2016 22:05:11 +0200

Hi,

I am currently looking at the checks an injecting server is supposed to 
perform regarding the syntax of header fields.

I am a bit confused by Appendix A.1 of RFC 3977, that gives rules for 
the format of Netnews articles:

A.1.  Header Folding

    Both email and Netnews articles are required to have at least one
    octet other than space or TAB on each header line.  Thus, folding can
    only happen at one point in each sequence of consecutive spaces or
    TABs.  Netnews articles are further required to have the header name,
    colon, and following space all on the first line; folding may only
    happen beyond that space.  Finally, some non-conforming software will
    remove trailing spaces and TABs from a line.  Therefore, it might be
    inadvisable to fold a header after a space or TAB.

    For maximum safety, header lines SHOULD conform to the following
    syntax rather than to that in Section 9.7.

      header = header-name ":" SP [header-content] CRLF
      header-content = [WS] token *( [CRLF] WS token )

The ABNF syntax, as well as the introductory text in Appendix A.1:
a/ allows a space as the only contents of a header field body;
b/ allows header field bodies with only a space in their first line;
c/ disallows trailing whitespace.

This syntax is given as a SHOULD.

If we have a look at Section 2.2 of RFC 5536, we have:

    o  All agents MUST generate header fields so that at least one space
       immediately follows the ':' separating the header field name and
       the header field body (for compatibility with deployed software,
       including NNTP [RFC3977] servers).  News agents MAY accept header
       fields that do not contain the required space.

    o  Every line of a header field body (including the first and any
       that are subsequently folded) MUST contain at least one non-
       whitespace character.

          NOTE: This means that no header field body defined by or
          referenced by this document can be empty.  As a result, rather
          than using the <unstructured> syntax from Section 3.2.5 of
          [RFC5322], this document uses a stricter definition:

    unstructured    =  *WSP VCHAR *( [FWS] VCHAR ) *WSP

So, mixed together, it means that a/ and b/ are finally disallowed, 
which is fine (the MUST of RFC 5536 is stronger that the SHOULD of RFC 
3977).
But c/ is still disallowed as a SHOULD.

It means that trailing whitespace SHOULD NOT be present in Netnews 
header field lines.  (RFC 3977 uses a stricter definition of 
"unstructured" than RFC 5536!)

Is it really a rule to enforce for Netnews articles?  And does someone 
know why this is not mentioned at all in RFC 5536?

-- 
Julien ÉLIE

« Life is short… so eat dessert first! »

---

From: Russ Allbery <[email-masked]>
Date: Sun, 07 Aug 2016 19:09:31 -0700

> know why this is not mentioned at all in RFC 5536?

I'm not sure how much we discussed this at the time, but normally trailing
whitespace is a SHOULD NOT generate because it tends to get mangled by
software and is very fragile.  But if that's the only justification (and I
can't think of another off-hand), it's not the sort of thing for a server
to enforce, just for a client to not do because it may not survive
transit.  If the server *can* handle it, it probably should try.

-- 
Russ Allbery ([email-masked])              <http://www.eyrie.org/~eagle/>

---

From: Julien ÉLIE <[email-masked]>
Date: Mon, 8 Aug 2016 22:45:37 +0200

Hi Russ,

> transit.  If the server *can* handle it, it probably should try.

OK, perfectly understood.
Let's just satisfy the "Duties of an Injecting Agent" (per RFC 5537), 
that require amongst other things to check the syntax defined by RFC 
5536.  It therefore does not include any checks about trailing whitespaces.

The general checks for headers are:
- line length (998 octets),
- no white-space-only lines in header field bodies,
- no control chars (except of course \r\n for folding, and \t),
- header name followed by colon+space and header body.

I believe these 4 checks can safely be implemented.
INN (nnrpd) already enforces the first.  It does not enforce the fourth, 
which is tolerated by RFC 5536 ("News agents MAY accept header fields 
that do not contain the required space.")  I believe it is OK to keep 
that tolerance.
However, the second and third checks should be enforced.  One can 
currently inject ^H in header names for instance.

I hope it corresponds to the rules USEFOR had in mind to enforce.

-- 
Julien ÉLIE