Subject: [Tools-arch] Draft RFP SoW for "Review of the current landscape of IETF document processing tools"

From: Jay Daley <[email-masked]>
Date: Thu, 25 Jun 2020 14:15:12 +1200

In support of the work of the Tools Architecture and Strategy Team (TAS), an RFP will be issued for a "Review of the current landscape of IETF document processing tools".  A draft SoW is below and I welcome your comments/feedback on it.

Jay

# Overview
The IETF seeks a contractor to review the current landscape of tools that have been specifically designed to be used by the IETF community to work with IETF documents (I-Ds and RFCs) and to build a dataset about these tools, including technical details, maintenance processes, licensing and others, and further to determine what parts of the document production, management and usage lifecycle each tool supports.

# Deliverables

## Dataset
This data is to be presented in a table using simple scales and classifications in addition to any supporting narrative.  For example, for licensing, a simple classification may include "GPL v3, Proprietary, BSD-like" and more.

The data to be collected is:

1.  A full catalog of tools, including all the relevant tools at https://tools.ietf.org and other tools that are not listed there but are known to be in use.   The contractor will be responsible for drawing up the list, which will then be reviewed, possibly iterated and approved.

2.  Details of the maintainer(s) of each tool

3.  The licensing of the tool (simple classification)

4.  The technical environment each tool is intended to work in (OS, packages, containers, etc)

5.  How each tool is maintained, including
- where the source code is located
- how issues are raised (simple classification)
- the level of version control and release management in use (on a simple scale)
- how comprehensive the test coverage is (on a simple scale) 
- the form of testing framework in use (simple classification)

6.  A subjective assessment of the current maintained state of the tool (on a simple scale)

6.  A subjective assessment of the suitability of each tool (on a simple scale), based on 3, 4, 5 and 6 above.

7.  What stages of the document lifecycle (see below) each tools covers.

## Document lifecycle
As part of this work, the contractor will need to construct a model of the IETF document lifecycle including document production (authoring, review, format conversion, etc), document access (download tools, search tools, etc) and document usage (extracting certain parts, etc).  This is only to cover the work carried out by the IETF community and not the work of the RFC Production Centre.

There are no templates for this model and it is expected that there will be two or three review and and change cycles.

## Coverage map
The information collected above is additionally to be presented in some form of map showing each tool and each stage, the coverage of each stage in the lifecycle and the subjective assessment of the suitability of each tool.

-- 
Jay Daley
IETF Executive Director
[email-masked]

---

From: "Martin Thomson" <[email-masked]>
Date: Thu, 25 Jun 2020 13:19:19 +1000

Thanks Jay,

I realize that while I might provide some of this list, I am completely unaware of the extent of it.  But this seems achievable in a relatively short time-scale.

I might prefer to see more details on how the contractor manages the work product.  Specifically, I think that this should be a relatively open process during which tool maintainers can see the assessment and make factual corrections.  A public document or repository might be a good way to ensure that contributions of this sort are easy - and getting those contributions might greatly simplify the process.

Is your intent to include the set of tools employed by the RPC in this?  I think that would be good, but it appears that you intentionally chose "used by the IETF community".

Nits:

Two items are numbered 6.

"and and" in "two or three review and and change cycles"

On Thu, Jun 25, 2020, at 12:15, Jay Daley wrote:
> In support of the work of the Tools Architecture and Strategy Team 
> (TAS), an RFP will be issued for a "Review of the current landscape of 
> IETF document processing tools".  A draft SoW is below and I welcome 
> your comments/feedback on it.
> 
> Jay
> 
> 
> # Overview
> The IETF seeks a contractor to review the current landscape of tools 
> that have been specifically designed to be used by the IETF community 
> to work with IETF documents (I-Ds and RFCs) and to build a dataset 
> about these tools, including technical details, maintenance processes, 
> licensing and others, and further to determine what parts of the 
> document production, management and usage lifecycle each tool supports.
> 
> # Deliverables
> 
> ## Dataset
> This data is to be presented in a table using simple scales and 
> classifications in addition to any supporting narrative.  For example, 
> for licensing, a simple classification may include "GPL v3, 
> Proprietary, BSD-like" and more.
> 
> 1.  A full catalog of tools, including all the relevant tools at 
> https://tools.ietf.org and other tools that are not listed there but 
> are known to be in use.   The contractor will be responsible for 
> drawing up the list, which will then be reviewed, possibly iterated and 
> approved.
> 
> 6.  A subjective assessment of the suitability of each tool (on a 
> simple scale), based on 3, 4, 5 and 6 above.
> 
> As part of this work, the contractor will need to construct a model of 
> the IETF document lifecycle including document production (authoring, 
> review, format conversion, etc), document access (download tools, 
> search tools, etc) and document usage (extracting certain parts, etc).  
> This is only to cover the work carried out by the IETF community and 
> not the work of the RFC Production Centre.
> 
> There are no templates for this model and it is expected that there 
> will be two or three review and and change cycles.
> 
> ## Coverage map
> The information collected above is additionally to be presented in some 
> form of map showing each tool and each stage, the coverage of each 
> stage in the lifecycle and the subjective assessment of the suitability 
> of each tool.
> 
> 
> 
> 
> -- 
> Jay Daley
> IETF Executive Director
> [email-masked]
> 
> -- 
> Tools-arch mailing list
> [email-masked]
> https://www.ietf.org/mailman/listinfo/tools-arch
>

---

From: Jay Daley <[email-masked]>
Date: Thu, 25 Jun 2020 15:59:33 +1200

> On 25/06/2020, at 3:19 PM, Martin Thomson <[email-masked]> wrote:
> 
> Thanks Jay,
> 
> I might prefer to see more details on how the contractor manages the work product.  Specifically, I think that this should be a relatively open process during which tool maintainers can see the assessment and make factual corrections.

That’s a good point yes - I will add that.

>  A public document or repository might be a good way to ensure that contributions of this sort are easy - and getting those contributions might greatly simplify the process.

I think if we limit it to tool maintainers contributing then that will work well

> Is your intent to include the set of tools employed by the RPC in this?  I think that would be good, but it appears that you intentionally chose "used by the IETF community".

My intent was to exclude tools that the community is not free to use.  I think the main tool the RPC use is xml2rfc and that is free for the community to use and so I would expect that to be included.  

Jay

>> [email-masked] <mailto:[email-masked]>
>> https://www.ietf.org/mailman/listinfo/tools-arch <https://www.ietf.org/mailman/listinfo/tools-arch>
>> 
> 
> -- 
> Tools-arch mailing list
> [email-masked] <mailto:[email-masked]>
> https://www.ietf.org/mailman/listinfo/tools-arch <https://www.ietf.org/mailman/listinfo/tools-arch>
-- 
Jay Daley
IETF Executive Director
[email-masked]

---

From: "John Levine" <[email-masked]>
Date: 25 Jun 2020 00:20:41 -0400

In article <[email-masked]> you write:
>> Is your intent to include the set of tools employed by the RPC in this?  I think that would be good, but it appears that you
>intentionally chose "used by the IETF community".
>
>My intent was to exclude tools that the community is not free to use.  I think the main tool the RPC use is xml2rfc and that is
>free for the community to use and so I would expect that to be included.  

The RPC use some commercial stuff such as the PDF tools from Callas.
It would be good to at least mention them since otherwise there'd be
mysterious gaps in how a document gets from A to B.

They also have a lot of custom web and command line stuff to manage
the publication process, e.g., wha document is in what state, what's
missinf refernces, and other odds and ends like the DOI uploader I
wrote. Some of that belongs to AMS, some (like the stuff I wrote) to
the IETF.

R's,
John

---

From: Jay Daley <[email-masked]>
Date: Fri, 26 Jun 2020 14:23:06 +1200

Thanks for all of your comments.  A revised version of the SoW is below, which will go to RFP unless I hear anything else.

Thanks to Russ for pointing out that these lists may include people who wish to bid for this RFP. and so please note that if this applies to you then you should either not provide any comment on the SoW, or declare that when providing comment. 

Jay

—— 
# SoW - Review of the current landscape of IETF document processing tools

## Overview

## Deliverables

### Deliverable 1 - Dataset

A dataset about the tools containing a combination of text, simple scales, simple classifications and supporting narrative.  For example, for licensing, a simple classification may include "GPL v3, Proprietary, BSD-like" and more.

The dataset is to be provided as raw data in CSV format.

The data to be collected for the dataset is:

1. A full catalog of tools, including all the relevant tools at https://tools.ietf.org and other tools that are not listed there but are known to be in use. 
2. Details of the maintainer(s) of each tool
3. The licensing of the tool (simple classification)
4. The technical environment each tool is intended to work in (OS, packages, containers, etc)
5. How each tool is maintained, including
    1. where the source code is located
    2. how issues are raised (simple classification)
    3. the level of version control and release management in use (on a simple scale)
    4. how comprehensive the test coverage is (on a simple scale) 
    5. the form of testing framework in use (simple classification)
6. A subjective assessment of the current maintained state of the tool (on a simple scale)
7. A subjective assessment of the suitability of each tool (on a simple scale), based on 3, 4, 5 and 6 above.
8. What stages of the document lifecycle (see below) each tool covers.

### Deliverable 2 -  Document lifecycle

A model of the IETF document lifecycle including document production (authoring, review, format conversion, etc), document access (download tools, search tools, etc) and document usage (extracting certain parts, etc).  This is only to cover the work carried out by the IETF community and not the work of the RFC Production Centre.

There are no templates for this model nor any existing work to build upon and so it is expected that this will be entirely new. 

### Deliverable 3 - Coverage map

Some form of coverage map/infographic showing each tool and each stage of the document lifecycle, the coverage of each stage in the lifecycle and the subjective assessment of the suitability of each tool.  This coverage map should allow at-a-glance understanding of the coverage of each stage and each tool including stages covered by multiple tools and stages not covered by any tool.

This coverage map is to be supplied in any common vector format.

## Requirements

### Selecting the tools to assess

The contractor will be responsible for drawing up the list of tools to be included for approval by the IETF LLC.

### Review and transparency

The IETF is committed to transparency and community engagement and so requires this work to be carried out as follows:

1. All three deliverables are to be developed using a public repository so that interested parties can see the work as it unfolds.
2. The initial list of tools proposed by the contractor will go out for community comment and potentially revised before the final version is presented for approval.
3. The maintainers of the tools are to be given every opportunity to recommend changes to the data and the assessments within (though the contractor retains the final say in what data they choose to include in the deliverable), which means:
    1. Any issues or pull requests raised by maintainers on the repository are to be considered and responded to.
    2. Once the initial version of the dataset is complete, it will go out for community comment and potentially revised.

### Work of the RFC Production Center

The RFC Production Center (RPC) is solely responsible for two steps in the document lifecycle:

1. The publication of the RFC, which uses a number of internal tools to manage references, assign DOIs and more. 
2. The conversion of the XML RFC into the long term ar PDF/A format for long term archive, which uses commercial software. 

The tools used solely for the two processes above are not within scope for this RFP but these steps should appear in the document lifecycle and coverage map.

## Additional Details

None.
—— 

-- 
Jay Daley
IETF Executive Director
[email-masked]

---

From: Jay Daley <[email-masked]>
Date: Fri, 26 Jun 2020 14:29:15 +1200

Sorry, some minor changes, please use this version and please remember that if you wish to bid for the RFP then you should either not provide any comment on the SoW, or declare that when providing comment. 

Jay

—— 

## Overview

## Deliverables

## Requirements

1. All three deliverables are to be developed using a public repository so that interested parties can see the work as it unfolds.
2. The initial list of tools proposed by the contractor will go out for community comment and the contractor will need to consider the feedback and potentially revise the list before the final version is presented for approval.
3. The maintainers of the tools are to be given every opportunity to recommend changes to the data and the assessments within (though the contractor retains the final say in what data they choose to include in the deliverable), which means:
    1. Any issues or pull requests raised by maintainers on the repository are to be considered and responded to.
    2. Once the initial version of the dataset is complete, it will go out for community comment and the contractor will need to consider the feedback and potentially revise the dataset before the final version is delivered.
4. The initial versions of the document lifecycle and coverage map will also go out for community feedback and the contractor will need to consider the feedback and potentially revise these before the final versions are delivered.

## Additional Details

None.

—— 

-- 
Jay Daley
IETF Executive Director
[email-masked]