Subject: [Tools-discuss] Data sources / alternatives to screen scraping

From: <[email-masked]>
Date: Mon, 15 Feb 2010 09:02:57 +0100

Hi all,

As Russ announced on the ietf-announce list a month ago, the upcoming
datatracker user interface changes will break scripts that do
"screen-scraping", or try to parse the HTML pages to extract some
information.

Much of the information is also available in forms that are intended
to parseable by scripts; however, those data sources aren't exactly
well documented (and it's not always easy to tell which of them are
authorative primary data, and which derived).

I've now updated the data source documentation on the Tools wiki
(http://trac.tools.ietf.org/group/tools/trac/wiki/DataSources).
Here's a quick summary about the most important primary data sources:

---------

Status of all internet-drafts (tab-separated text; generated from the
database by a cron job once a day):
http://www.ietf.org/id/all_id.txt

Title/authors/abstract/date for active internet-drafts (text;
generated from the database by cron jobs once a day. Although these
text files were probably originally intended for mainly human
consumption, their format has been very stable over the years, and as
they're currently used by number of tools, I would expect this to
continue):
http://www.ietf.org/id/1id-index.txt
http://www.ietf.org/id/1id-abstracts.txt

All RFCs (XML/text; note that the XML has much more information than
the text version):
http://www.rfc-editor.org/rfc/rfc-index.xml
http://www.rfc-editor.org/rfc/rfc-index.txt

Documents that are currently in IETF last call (Atom feed; generated
from the database on-the-fly):
http://datatracker.ietf.org/feed/last-call/

Document that are on the agenda of upcoming IESG telechats
(tab-separated text; generated from the database on-the-fly):
http://datatracker.ietf.org/iesg/agenda/documents.txt

Documents in the RFC editor queue (XML):
http://www.rfc-editor.org/queue2.xml

Detailed datatracker history for a particular draft (Atom feed;
generated from the database on-the-fly):
http://datatracker.ietf.org/feed/comments/draft-ietf-msec-newtype-keyid/

Information about active WGs, chairs, mailing lists, charters, etc
(text; generated from the database on-the-fly):
http://datatracker.ietf.org/wg/summary.txt
http://datatracker.ietf.org/wg/1wg-charters.txt

IPR disclosures by draft (tab-separated text; generated from the
database on-the-fly):
http://datatracker.ietf.org/ipr/by-draft/ 

---------

The upcoming datatracker UI changes will affect only HTML pages,
not any of the text/Atom/etc. URLs listed above.
 
Best regards,
Pasi