Subject: [Sidrops] Manifest entry filename validation

From: Erik Rozendaal <[email-masked]>
Date: Thu, 19 Nov 2020 10:03:53 +0100

Hi all,

During the past months we've been working on improving the RPKI 
validator to track the 6468bis RFC. As part of that effort we've also
been finding and fixing various other mismatches with the current
RPKI RFCs.

One part of this is validating manifest entry file names. Currently the
various validator implementations handle manifest filename entries
differently, potentially resulting in different validation results.

Summary:

We think the manifest RFC 6486 should define rules used for the filename
entries in a manifest.

Our proposal is to only allow a minimal set of
characters from the ASN.1 IA5String type: a-z, A-Z, 0-9, . (dot), -
(dash), _ (underscore).  Furthermore blank entries, ".", and ".." must
not be allowed. Filename extensions should be matched in a case
insensitive manner when determining object type (ROA, CRL, etc).

These rules validate all current objects from the major trust anchors
(all RIRs and APNIc AS0). They avoid special URI characters and
characters that may be used to navigate file system directories.

We may also want to add rules such as:

- Avoid illegal (Windows) filenames such as PRN, NUL, or CON (see
 https://docs.microsoft.com/en-us/windows/win32/fileio/naming-a-file <https://docs.microsoft.com/en-us/windows/win32/fileio/naming-a-file>).

- Require that all entries have a three letter filename extension.

- Prohibit entries that only differ by upper or lower case (FOO.CER vs
 foo.cer).

Background:

Manifest entry filename validation

Currently the different RPKI validators use different rules to
validate the filename of a manifest entry. The RFC says (section
4.2.1 under "fileList"):

"Each FileAndHash is an ordered pair consisting of the name of the
file in the repository publication point (directory) that contains
the object in question and a hash of the file's contents."

Unfortunately it is not exactly clear what is allowed as the
filename. The three validators that I investigated use different
strategies:

rpki-validator-3: anything goes. Filename is resolved using the
base repository URI and the `resolve` method on the Java URI
class. This allows subdirectories, escaping to parent directories,
etc. This will be fixed by
https://github.com/RIPE-NCC/rpki-commons/pull/30 <https://github.com/RIPE-NCC/rpki-commons/pull/30> where a filename
must:

- only have characters a-z, A-Z, 0-9, <dot>, (, ), +, -, ', :, =, or _.
- not be blank
- not be "."
- not be ".."

All currently published objects pass these test.

rpki-client: filename entries must be at least four characters and
the filename should not contain a forward slash (/). See
https://cvsweb.openbsd.org/src/usr.sbin/rpki-client/mft.c?rev=1.14.4.1&content-type=text/x-cvsweb-markup <https://cvsweb.openbsd.org/src/usr.sbin/rpki-client/mft.c?rev=1.14.4.1&content-type=text/x-cvsweb-markup>
in the `mft_parse_filehash` function.

routinator: filename cannot have control characters or spaces, ", #,
<, >, ?, [, \, ], ^, `, {, |, } or have value >= 0x7f (see
https://github.com/NLnetLabs/rpki-rs/blob/b3b0eb6b0c524d1bdd8de23acc6cf3988386b3b0/src/uri.rs#L682 <https://github.com/NLnetLabs/rpki-rs/blob/b3b0eb6b0c524d1bdd8de23acc6cf3988386b3b0/src/uri.rs#L682>).
Furthermore "."  and ".." are not allowed as segment anywhere in a
complete rsync URI (see
https://github.com/NLnetLabs/rpki-rs/blob/b3b0eb6b0c524d1bdd8de23acc6cf3988386b3b0/src/uri.rs#L95 <https://github.com/NLnetLabs/rpki-rs/blob/b3b0eb6b0c524d1bdd8de23acc6cf3988386b3b0/src/uri.rs#L95>).

Kind regards,
Erik

---

From: Stephen Kent <[email-masked]>
Date: Thu, 19 Nov 2020 09:59:48 -0500

Erik,

Thanks for looking into this problem and proposing a solution. I'm 
anxious to incorporate changes into the -bis version before the end of 2020.

I believe the character set restrictions are appropriate, as is 
prohibiting single and double period directory entry names, and 
mandating the presence of the 3-character suffix. I also have no problem 
with prohibiting a specific set of names, such as the Windows-centric 
ones you mention.

The proposal to prohibit entries that differ only due to capitalization 
is worrisome. All of the other syntactic tests are easy to perform on 
individual entries. This test seems to require examining all entries at 
once to detect a violation. I'd rather not include this rule, because of 
the added complexity. Would folks like to mandate use of only upper or 
lower case characters as an alternative?

Steve

---

From: Erik Rozendaal <[email-masked]>
Date: Thu, 19 Nov 2020 16:53:36 +0100

Hi Steve,

> On 19 Nov 2020, at 15:59, Stephen Kent <stkent=[email-masked]> wrote:
> 
> I believe the character set restrictions are appropriate, as is prohibiting single and double period directory entry names, and mandating the presence of the 3-character suffix. I also have no problem with prohibiting a specific set of names, such as the Windows-centric ones you mention.
> 
> The proposal to prohibit entries that differ only due to capitalization is worrisome. All of the other syntactic tests are easy to perform on individual entries. This test seems to require examining all entries at once to detect a violation. I'd rather not include this rule, because of the added complexity. Would folks like to mandate use of only upper or lower case characters as an alternative?

Most of the additional proposed rules are an (probably incomplete) attempt to make rsync work on non-Unix systems. Maybe we can just make this a recommendation or SHOULD, or just forget about this. RRDP should not be affected unless you directly map object URIs to filenames on a file system. 

Only allowing lower- or uppercase characters will break existing objects in the RPKI and require changes at the CA level, so that's probably a no-go.

Kind regards,
Erik

---

From: Job Snijders <[email-masked]>
Date: Thu, 19 Nov 2020 16:47:03 +0000

On Thu, Nov 19, 2020 at 10:03:53AM +0100, Erik Rozendaal wrote:
> Summary:
> 
> We think the manifest RFC 6486 should define rules used for the
> filename entries in a manifest.

I dont think one would want to go as far as to "make rpki work on case
insensitive filesystems like some versions of macosx or windows".

RPKI objects map to RPKI files. Paths and filenames are considered from
the IEEE Std 1003.1-2008 / POSIX.1 interface perspective to improve
portability.

Filesystems with with an assortment of capability limitations (example
FAT32's case insensitivity or small filename length limits) just aren't
suitable.

> Our proposal is to only allow a minimal set of characters from the
> ASN.1 IA5String type: a-z, A-Z, 0-9, . (dot), - (dash), _
> (underscore).  Furthermore blank entries, ".", and ".." must not be
> allowed. Filename extensions should be matched in a case insensitive
> manner when determining object type (ROA, CRL, etc).

Or alternatively the spec requires all RPKI filename extensions to be
lower case? At the moment of writing all published rpki files seem to
use lower case anyhow.

>  https://docs.microsoft.com/en-us/windows/win32/fileio/naming-a-file

I consider attempting to protect against windows-specific bugs very much
out of scope for this working group :-)

> - Require that all entries have a three letter filename extension.

ack

> - Prohibit entries that only differ by upper or lower case (FOO.CER vs

Requiring filename extensions to be lowercase addresses some of this,
but 'FOO.cer' and 'foo.cer' really are separate paths.

Regards,

Job

---

From: Tim Bruijnzeels <[email-masked]>
Date: Thu, 19 Nov 2020 17:47:27 +0100

Hi,

> On 19 Nov 2020, at 16:53, Erik Rozendaal <[email-masked]> wrote:
> 
> Hi Steve,
> 

Indeed this would be hard for my CA implementation. Or well, it's easy enough to fix in future, but there are versions deployed today that use both lower and uppercase.

So, where was this coming from? Case-preserving, but insensitive, file systems?

I could live with some normative language that instructs a CA to avoid creating case-insensitive collisions between filenames.

Tim

> 
> Kind regards,
> Erik
> 
> _______________________________________________
> Sidrops mailing list
> [email-masked]
> https://www.ietf.org/mailman/listinfo/sidrops

---

From: Stephen Kent <[email-masked]>
Date: Thu, 19 Nov 2020 12:32:42 -0500

How about the following text:

Section 4.2.2

To promote interoperability across file systems, the following 
restrictions are imposed on the file names that appear in the fileList:

1.Only a subset of IA5string characters are permitted: a-z, A-Z, 0-9, . 
(dot), - (dash), and _ (underscore).

2.No blank file names are permitted.

3.Single and double dot (. and ..) file names are prohibited.

4.The three-character filename extension defined for each object type 
MUST be present and MUST be lowercase, e.g., cer, roa, and crl.

---

From: Job Snijders <[email-masked]>
Date: Thu, 19 Nov 2020 18:00:34 +0000

On Thu, Nov 19, 2020 at 12:32:42PM -0500, Stephen Kent wrote:
> To promote interoperability across file systems, the following restrictions
> are imposed on the file names that appear in the fileList:
> 
> 4.The three-character filename extension defined for each object type MUST
> be present and MUST be lowercase, e.g., cer, roa, and crl.

I reworded things a bit:

-----------

The following restrictions are imposed on the names that appear in the fileList:

1. Only a subset of IA5string characters are permitted: a-z, A-Z, 0-9, . (DOT), - (HYPHEN), and _ (UNDERSCORE).

2. Each name MUST be suffixed with the appropriate filename extension defined the object type and MUST be lowercase, e.g., '.cer', '.roa', or '.crl'.

3. The . (DOT) character MUST appear exactly once in a name.

---

From: Job Snijders <[email-masked]>
Date: Thu, 19 Nov 2020 20:27:38 +0000

> > How about the following text:

now with less typos:

--------------------

Section 4.2.2 "Names in FileAndHash objects"

2. Each name MUST be suffixed with the appropriate filename extension defined for the object type, and MUST be lowercase, e.g., '.cer' or '.roa'.

3. The . (DOT) character MUST appear exactly once.

As an example, 'vixxBTS_TVXQ-2pmGOT7.cer' is a valid filename.

---

From: Stephen Kent <[email-masked]>
Date: Mon, 23 Nov 2020 09:43:40 -0500

Job,

I like your more concise phrasing of the file name restrictions. I note, 
however, that they are not identical to what was initially proposed. For 
example, the original proposal did not prohibit more than one "." in a 
file name, except for the ".." case.

Before I adopt your suggested text I'd like to see comments from the WG 
that confirm agreement with your version of the constraints.

Thanks,

Steve

---

From: Ties de Kock <[email-masked]>
Date: Mon, 23 Nov 2020 16:00:31 +0100

Hi,

> On 19 Nov 2020, at 21:27, Job Snijders <[email-masked]> wrote:
> 

A request for clarification: Do you intend this to convey that filenames
have to be completely lower-case?

The current phrasing is slightly ambiguous and can be read to include "each name
MUST be lowercase", while the examples show that the extension must be lower-case.

Kind regards,
Ties

---

From: Job Snijders <[email-masked]>
Date: Mon, 23 Nov 2020 15:10:49 +0000

Thanks Ties, how about:

--------------------

The following restrictions are imposed on the names that appear in the
fileList:

2. Each name MUST be suffixed with the appropriate filename extension defined for the object type, and the filename extension MUST be lowercase, e.g., '.cer' or '.roa'.

---

From: Russ Housley <[email-masked]>
Date: Mon, 23 Nov 2020 10:21:56 -0500

If no one is useing multiple DOTs in their filenames, this this is clear and straightforward.

Russ

> On Nov 23, 2020, at 9:43 AM, Stephen Kent <stkent=[email-masked]> wrote:
> 
> Job,
> 
> I like your more concise phrasing of the file name restrictions. I note, however, that they are not identical to what was initially proposed. For example, the original proposal did not prohibit more than one "." in a file name, except for the ".." case.
> 
> Before I adopt your suggested text I'd like to see comments from the WG that confirm agreement with your version of the constraints.
> 
> Thanks,
> 
> Steve

---

From: Di Ma <[email-masked]>
Date: Tue, 24 Nov 2020 13:33:50 +0800

Steve,

I am in favor of your suggestion, based on our experience of developing and maintaining RP software .

Di

> 2020年11月20日 01:32，Stephen Kent <stkent=[email-masked]> 写道：
> 
> ï¿½
> To promote interoperability across file systems, the following restrictions are imposed on the file names that appear in the fileList:
> ï¿½
> 1. Only a subset of IA5string characters are permitted: a-z, A-Z, 0-9, . (dot), - (dash), and _ (underscore).
> 2. No blank file names are permitted.
> 3. Single and double dot (. and ..) file names are prohibited.
> 4. The three-character filename extension defined for each object type MUST be present and MUST be lowercase, e.g., cer, roa, and crl.

---

From: Tim Bruijnzeels <[email-masked]>
Date: Tue, 24 Nov 2020 09:21:20 +0100

Hi Job, Steve, all,

I am losing track a bit, but I believe this text below was the latest suggestion by Job to Steve, right?

Question / suggestion on extensions below.

> On 23 Nov 2020, at 16:10, Job Snijders <[email-masked]> wrote:
> 
> Thanks Ties, how about:
> 
> --------------------
> 

I think we should probably have a normative reference here to section 7.2 of RFC 6481 (Repository Structure), or better yet to the "RPKI Repository Naming Scheme" registry maintained by IANA:

https://www.iana.org/assignments/rpki/rpki.xhtml

As that section says the extensions are supposed to be "three-letter", presumably to make file name parsing safe. Not sure if that needs to be repeated here explicitly.

As a side note, while I am here.. I would also like to note that it's a bit unfortunate that '.cer' can be _any_ resource certificate. So it could be:
- An RPKI CA certificate used for delegations
- A 'naked' EE certificate (as possibly used by RFC 7909)
- A BGPSec Router certificate

At least my quick scanning of RFC 6481 seems to imply this, as it uses the term 'certificate' without applying further constraints. However, it would be nicer for RPs if these cases used different extensions. This could simplify parsing quite a bit. And while this would be a breaking change in theory, I think it could be done without impact in practice since currently *all* .cer files found in the wild are CA certificates.

> As an example, 'vixxBTS_TVXQ-2pmGOT7.cer' is a valid filename.

Looks good to me.

Tim

---

From: Ties de Kock <[email-masked]>
Date: Tue, 24 Nov 2020 14:20:40 +0100

Hi Job, all,

> On 23 Nov 2020, at 16:10, Job Snijders <[email-masked]> wrote:

Looks good. However, after an internal discussion during which we realised that
empty names (e.g. .cer) were valid filenames under the previous proposal, and
after validating that all current valid objects from the major trust anchors
adhere to our proposal, we would like to propose:

----------

Names that appear in the fileList MUST consist of one or more characters from
the set a-z, A-Z, 0-9, - (HYPHEN), or _ (UNDERSCORE), followed by a single .
(DOT), followed by a three-letter extension a-z.

---

From: Robert Kisteleki <[email-masked]>
Date: Tue, 24 Nov 2020 14:29:25 +0100

> adhere to our proposal, we would like to propose:

At this point perhaps a regexp would be simpler to specify?

I also note that both miss a clause saying the filenames are case 
sensitive, thus 'vixxBTS_TVXQ-2pmGOT7.cer' and 
'VixxBTS_TVXQ-2pmGOT7.cer' are different certificates.

Robert

---

From: Stephen Kent <[email-masked]>
Date: Tue, 24 Nov 2020 09:42:46 -0500

Ties,

I think we're getting closer to a rigorous description. I like your 
concise text, but it doesn't cite the relevant list of allowed 
extensions, as suggested by Tim.  Also, the requirement that extensions 
be lowercase is redundant - RFC 6481
already dictates this for manifests, ROAs, CRLs, GB records, and  (all 
types of) certs.

Thus I suggest the following:

Names that appear in the fileList MUST consist of one or more characters chosen
from the set a-z, A-Z, 0-9, - (HYPHEN), or _ (UNDERSCORE), followed by a single .
(DOT), followed by a three-letter extension. The extension MUST be one of those
enumerated in the "RPKI Repository Naming Scheme" registry maintained by IANA [IANA]

add the following Normative Reference entry:

[IANA] https://www.iana.org/assignments/rpki/rpki.xhtml#name-schemes

---

From: Job Snijders <[email-masked]>
Date: Tue, 24 Nov 2020 15:03:29 +0000

On Tue, Nov 24, 2020 at 09:42:46AM -0500, Stephen Kent wrote:
> I think we're getting closer to a rigorous description. I like your concise
> text, but it doesn't cite the relevant list of allowed extensions, as
> suggested by Tim.  Also, the requirement that extensions be lowercase
> is redundant - RFC 6481 already dictates this for manifests, ROAs,
> CRLs, GB records, and  (all types of) certs.
> 

OK for me, thank you all for picking at this until it looked right

Kind regards,

Job

---

From: Russ Housley <[email-masked]>
Date: Tue, 24 Nov 2020 10:12:02 -0500

Tim:

RFC 2585 is the document that specifies the conventions of ".cer" and ".crl" for X.509 certificates and X,509 CRLs, respectively.

Russ

> On Nov 24, 2020, at 3:21 AM, Tim Bruijnzeels <[email-masked]> wrote:
> 
> Hi Job, Steve, all,
>

---

From: Tim Bruijnzeels <[email-masked]>
Date: Tue, 24 Nov 2020 16:44:44 +0100

Hi Russ,

> On 24 Nov 2020, at 16:12, Russ Housley <[email-masked]> wrote:
> 
> Tim:
> 

That may well be, but I still think it would be useful, in the context of RPKI, to be able to differentiate between different types of .cer *before* parsing rather than differentiate while parsing. And if we did, then an entry in the IANA registry "RPKI Repository Naming Scheme" could be appropriate. Anyway, I don't want to complicate things - if RP implementers don't speak up that they want this, then I would say we just leave it as an aside..

The other more important reason for bringing up that registry is because it already defines the extensions one can expect. So I would suggest that the following in item 2

  extension defined for the object type

Becomes:

  extension defined for the object type in the IANA registry "RPKI Repository Name Scheme" 

Tim

---

From: Stephen Kent <[email-masked]>
Date: Mon, 30 Nov 2020 15:21:57 -0500

I've seen no responses to Tim's suggestion to create alternate extension 
types for certs used to other purposes, e.g., router certs, so I will 
ask my buddies to issue another version of the ID with the text below. 
Thanks to everyone for their contributions to this additional, 
clarifying text.

Steve

---

From: Tim Bruijnzeels <[email-masked]>
Date: Tue, 1 Dec 2020 08:25:04 +0100

Hi Steve,

> On 30 Nov 2020, at 21:21, Stephen Kent <stkent=[email-masked]> wrote:
> 
> I've seen no responses to Tim's suggestion to create alternate extension types for certs used to other purposes, e.g., router certs, so I will ask my buddies to issue another version of the ID with the text below. Thanks to everyone for their contributions to this additional, clarifying text.

Works for me. If we should ever think about updating the manifest structure (out of scope for now clearly), then we could consider to include specific type information fields rather than depending on filename extensions.

Tim