Subject: #1028 USEFOR 3.1.2: What zones should be on the MUST accept list?

From: Harald Tveit Alvestrand <[email-masked]>
Date: Wed, 08 Jun 2005 13:24:11 +0200

I've created the following ticket:

USEFOR section 3.1.2 "Date" currently says:

3.1.2  Date

   The Date header is the same as that specified in Sections 3.3 and
   3.6.1 of [RFC2822] with the added restrictions detailed in
   Section 2.2.  However, the use of "UT" and "GMT" as time zones
(part
   of <obs-zone>), although deprecated, is widespread in news articles
   today.  Therefore, agents MUST accept <date-time> constructs which
   include the updated <zone> construct below.

   orig-date       =  "Date:" SP date-time CRLF

   zone            =  (( "+" / "-" ) 4DIGIT) / "UT" / "GMT"

   Note that agents SHOULD NOT generate <date-time> constructs which
   include either "UT" or "GMT" and MUST NOT generate <date-time>
   constructs which include any other zone names defined by
<obs-zone>,
   some of which have ambiguous interpretations and would have adverse
   effects on the Netnews protocols.

   Also note that these requirements apply wherever <date-time> is used,
   including Injection-Date and Expires in Section 3.2.1 and
   Section 3.2.4 respectively.

The question has been raised about how realistic this is.
Russ Allbery computed the following numbers:

I was curious, so here are some actual numbers.

% cd spool/overview
% find -name \*.DAT -print > /tmp/files
% xargs cut -f 4 < /tmp/files | sed 's/ (.*)$//' > /tmp/dates
% wc -l /tmp/dates
3632585 /tmp/dates
% foreach zone ( GMT UT UTC EDT MDT CDT PDT )
foreach? echo -n "${zone}: "
foreach? grep $zone'$' /tmp/dates | wc -l
foreach? end
GMT:  780813
UT:       13
UTC:    3756
EDT:   30721
MDT:       1
CDT:      21
PDT:     216

Which is more than I thought.  MDT, CDT, and PDT are a lot less than I
thought.  Apparently someone does actually use UT, but maybe only one
person.  :)

These numbers could argue for skipping UT and/or including EDT (and 
presumabily EST, since this measurement was taken in summer).

(note - not in ticket - last sentence was added by me, not Russ)

---

From: Henry Spencer <[email-masked]>
Date: Wed, 8 Jun 2005 10:27:23 -0400 (EDT)

> presumabily EST, since this measurement was taken in summer).

"UT" was my addition in son-of-1036, and since it seems to have achieved
essentially zero market penetration :-), I think we can forget about it. 

We should *NOT* bless EST or EDT, still in significant use though they may
be, because they are definitely ambiguous.  (I think it most unlikely that
software which puts in a local timezone name is first checking to make
sure that the definition agrees with the RFC 2822 obs-zone definition.)

                                                          Henry Spencer
                                                       [email-masked]

---

From: Bruce Lilly <[email-masked]>
Date: Wed, 8 Jun 2005 19:20:17 -0400

> Russ Allbery computed the following numbers:
[...]
> presumabily EST, since this measurement was taken in summer).

That reminds me; the corresponding [ECMP]ST (case-insensitive) versions
should probably also be counted.  Some places (e.g. the U.S. state of
Indiana) do not observe daylight savings time, and some software is so
broken that it uses the wrong zones, or fails to take account of (or is
configured to fail to take account of) DST.  Indiana, incidentally, uses
EST year-round.

---

From: Russ Allbery <[email-masked]>
Date: Wed, 08 Jun 2005 17:16:00 -0700

> configured to fail to take account of) DST.  Indiana, incidentally, uses

Okay, okay, fine.  :)  #JUNK# are various date lines that could not be
parsed into a time zone by my simple algorithm (most of them didn't
specify a time zone at all, but some of them had other problems, like not
zero-padding the seconds in the timestamp).

Hopefully this will give people a new appreciation for just how much
bizarre junk is out there on the wire.

      #JUNK#:    1553 (00.0%)
      +-0400:      10 (00.0%)
        +000:       8 (00.0%)
       +0000:  165048 (04.5%)
       +0001:       1 (00.0%)
       +0028:       1 (00.0%)
       +0100:  190268 (05.2%)
       +0102:       1 (00.0%)
       +0200:  563629 (15.3%)
       +0300:   12204 (00.3%)
       +0307:       1 (00.0%)
       +0400:    5279 (00.1%)
       +0430:      21 (00.0%)
       +0500:     477 (00.0%)
       +0530:     881 (00.0%)
       +0545:     421 (00.0%)
       +0600:     675 (00.0%)
       +0630:       1 (00.0%)
       +0700:    1174 (00.0%)
       +0800:   39837 (01.1%)
       +0900:    7291 (00.2%)
  +0900(KST):       1 (00.0%)   # This is legit, just a flaw in my script
       +0930:    2695 (00.1%)
       +0950:      77 (00.0%)
        +100:       2 (00.0%)
       +1000:   29806 (00.8%)
       +1030:       3 (00.0%)
       +1100:      75 (00.0%)
       +1200:    8782 (00.2%)
       +1300:       5 (00.0%)
       +1400:      18 (00.0%)
       -0000:   18257 (00.5%)
       -0001:      25 (00.0%)
       -0002:       1 (00.0%)
       -0100:      64 (00.0%)
       -0200:      63 (00.0%)
       -0230:     461 (00.0%)
       -0300:    4596 (00.1%)
       -0400:  262862 (07.1%)
       -0500:  122173 (03.3%)
       -0600:   29009 (00.8%)
       -0700: 1116599 (30.3%)
       -0756:       1 (00.0%)
       -0800:  269930 (07.3%)
       -0900:      13 (00.0%)
       -1000:     684 (00.0%)
       -1100:       1 (00.0%)
       -1200:      45 (00.0%)
        -500:       6 (00.0%)
        BRST:       2 (00.0%)
         BST:      59 (00.0%)
         CDT:      22 (00.0%)
        CEST:      18 (00.0%)
         CET:       7 (00.0%)
         CST:    4224 (00.1%)
     EASTERN:       1 (00.0%)
         EDT:   31806 (00.9%)
         EET:       1 (00.0%)
         EST:      40 (00.0%)
          GM:      84 (00.0%)
         GMT:  782196 (21.3%)
        GMT):       7 (00.0%)
    GMT+0000:      35 (00.0%)
       GMT+1:      65 (00.0%)
       GMT+2:       1 (00.0%)
       GMT+6:      60 (00.0%)
       GMT-6:       3 (00.0%)
         JST:       3 (00.0%)
       LOCAL:       3 (00.0%)
         MDT:       1 (00.0%)
        MEST:       1 (00.0%)
         MET:       6 (00.0%)
         MEZ:      56 (00.0%)
         MST:       3 (00.0%)
         NZS:       1 (00.0%)
         PDT:     214 (00.0%)
         PST:    2834 (00.1%)
   UNDEFINED:       9 (00.0%)
          UT:      13 (00.0%)
         UTC:    3733 (00.1%)
¥X¥_¼Ð·Ç®É¶¡:       1 (00.0%)

This data was collected with:

find . -name \*.DAT | xargs cut -f 4 \
    | sed -e 's/.*:[0-9][0-9] //' -e 's/ (.*//' \
    | perl -n /tmp/script

where /tmp/script is:

chomp;
if (\!$_ || / /) {
    $_ = "#junk#";
}
$zones{uc $_}++;
$total++;
END {
    for (sort keys %zones) {
        printf "%12s: %7d (%04.1f%%)\n", $_, $zones{$_},
            $zones{$_} / $total * 100
    }
}

I write better code when I expect to ever use it again.  :)

The large numbers of PST compared to PDT is probably an artifact of my
spool retention; I have a bunch of local newsgroups that I keep forever,
I'm in the Pacific time zone, and I'm at a university where almost all of
our traffic is not during daylight saving time.

-- 
Russ Allbery ([email-masked])             <http://www.eyrie.org/~eagle/>

---

From: "Charles Lindsey" <[email-masked]>
Date: Thu, 9 Jun 2005 16:32:53 GMT

>sure that the definition agrees with the RFC 2822 obs-zone definition.)

I concur.

GMT is a special case because (a) it is not ambiguous and (b) it occurs in
20% of current articles (presumably because some widely used software is
enforcing it - I know that CNews is one of the culprits in that regard).

So I think we MUST accept GMT, though I would happily drop UT.

As for the rest, I do not think we should cherry pick.

Either we accept everything in the RFC 2822 <obs-zone> (with some risk of
misuse in Australia and other places), or we accept none of it beyond
GMT. My preference is the latter, followed by the former (OK, you might
omit the military ones).

As for generation, we currently say SHOULD NOT. I would have no probnlem
if someone wants to promoite that to MUST NOT.

BTW, I discovered this morning that "EST" is also used in Eastern Brazil
and Easter Island :-) .

-- 
Charles H. Lindsey ---------At Home, doing my own thing------------------------
Tel: +44 161 436 6131 Fax: +44 161 436 6133   Web: http://www.cs.man.ac.uk/~chl
Email: [email-masked]      Snail: 5 Clerewood Ave, CHEADLE, SK8 3JU, U.K.
PGP: 2C15F1A9      Fingerprint: 73 6D C2 51 93 A0 01 E7 65 E8 64 7E 14 A4 AB A5

---

From: Frank Ellermann <[email-masked]>
Date: Fri, 10 Jun 2005 03:09:50 +0200

Russ Allbery wrote:

> Hopefully this will give people a new appreciation for just
> how much bizarre junk is out there on the wire.

Thanks, nice statistics.  So nobody uses Z etc. ?

>          MEZ:      56 (00.0%)

Ugh.  But at least no MESZ or "MEZT".

> I write better code when I expect to ever use it again.  :)

I won't understand it unless it's gawk ;-)  If you're sure
that you didn't miss the military zones I simply believe it.

For USEFOR your results mean "no GMT hits 21%", that would be
bad.  Henry already said that he won't insist on UT.  So the
real problem is EDT + EST (0.9%).
                                 Bye, Frank

---

From: Russ Allbery <[email-masked]>
Date: Thu, 09 Jun 2005 19:21:04 -0700

> Thanks, nice statistics.  So nobody uses Z etc. ?

I've never seen a use of the military time zones in the wild, although
it's possible my script might have missed some uses if there was no space
before the time zone name (which is allowed by RFC 822; RFC 822 doesn't
require any spaces at all in the string, last time I checked).  But I
doubt it.

> Ugh.  But at least no MESZ or "MEZT".

At least in this batch of stuff!  :)

Incidentaly, I think I checked at one point and the LOCAL and UNKNOWN time
zones were from articles from the 1990s that were in groups that never
expired.  I'm not sure, though.

> real problem is EDT + EST (0.9%).

I think it would be insane to allow anything other than what's in RFC 822.
I'm certainly not arguing for adding BST, JST, etc.  I'm mostly pointing
out two things:  EDT/EST/PDT/PST/etc. are both fully compliant with RFC
1036 and used in practice by significant numbers of articles
(*particularly* if you want to do things with old news spools, which
people do often want to do), and real-life news software accepts a ton of
junk that's never been in any standard because of common use of getdate as
mentioned in RFC 1036.  getdate supports an unfortunate number of textual
time zone strings.

The Date specification in RFC 1036 is unfortunately something of a mess.

---

From: Graham Drabble <[email-masked]>
Date: Fri, 10 Jun 2005 11:29:22 +0100

On 09 Jun 2005 Russ Allbery <[email-masked]> wrote in
news:[email-masked]: 

> Hopefully this will give people a new appreciation for just how
> much bizarre junk is out there on the wire.

I came accross a whole bunch of articles that had the form:

Tue, 26 Apr 2005 21:33:30 +Z

This was due to a badly configured moderation bot so it was affecting 
all the posts into the group.

FWIW this doesn't seem to have affected propogation and my newsreader 
(XNews) gets the date right and treats +Z as +0000. My stat gathering 
tool which uses perl's Date::Parse broke.

-- 
Graham Drabble
http://users.ox.ac.uk/~sjoh1646/

---

From: Russ Allbery <[email-masked]>
Date: Fri, 10 Jun 2005 09:38:14 -0700

Graham Drabble <[email-masked]> writes:
> On 09 Jun 2005 Russ Allbery <[email-masked]> wrote:

> all the posts into the group.

They were so close.  Without the +, that would have actually been valid.

> tool which uses perl's Date::Parse broke.

INN treats all time zones it can't parse as the local time zone, rather
than +0000, just because it did that historically and when I wanted to
change it to +0000, again, people objected.

---

From: Graham Drabble <[email-masked]>
Date: Sat, 11 Jun 2005 00:48:28 +0100

>> This was due to a badly configured moderation bot so it was
>> affecting all the posts into the group.
> 

Apparantly they were missing a '%' and had the wrong case Z.

The point that I was trying to make is that any software that is 
actually going to work on Usenet needs to be able to cope with pretty 
much anything in there. Trying to define what all the possible things 
that could be used is going to be futile so we are probably best off 
keeping the list short and saying something about how to handle the 
crap.

My suggestion is to have all letters as SHOULD NOT generate (there's no 
need for them, the numeric format works just as well if not better) and 
then don't explicitly say how to treat them. Wa can then say that any 
time with an unparceable offset (either because they use GMT, UT, EDT 
etc or because they've mangled it in some new way) should be treated as 
+0000. This means that GMT (which appears to be the most common) still 
gets parsed the same way and junk is handled consistently.