Subject: Re: [IPFIX] [OPS-DIR] FW: Evaluation: draft-ietf-ipfix-file-03.txt to Proposed Standard

From: Margaret Wasserman <[email-masked]>
Date: Tue, 14 Apr 2009 11:18:01 -0400

I have some significant concerns about draft-ietf-ipfix-file-03.txt  
that I would like to discuss.

ISSUE #1:  Security Requirements

This document does not seem to be consistent with its own stated  
security requirements, nor does it specify any interoperable security  
mechanisms.  In particular, sections 5.6 of the document says:
"5.6. Creator Authentication and Confidentiality
"Archival storage of flow data may also require assurance that no  
unauthorized entity can read or modify the stored data. Asymmetric-  
key cryptography can be applied to this problem, by signing flow data  
with the private key of the creator, and encrypting it with the public  
keys of those authorized to read it. The ideal flow storage format  
will support the encryption and signing of flow data. As with error  
correction, this problem has been addressed well at a layer below that  
addressed by this document. Instead of specifying a particular choice  
of encryption technology, we can leverage the fact Trammell, et al.  
Expires April 27, 2009 [Page 12]  Internet-Draft IPFIX Files October 20 
08 that existing cryptographic technologies work quite well on data  
stored in files to meet this requirement. Beyond support for the use  
of TLS for transport over TCP or DTLS for transport over SCTP or UDP,  
both of which provide transient authentication and confidentiality,  
the IPFIX protocol does not support this requirement directly. It is  
assumed that this requirement will be met externally."
I agree with this section, but I cannot find any place in the document  
where it describes what type of file signing technology should be  
implemented in File Readers and Writers, what the certificates would  
look like and/or how they would be bound to the file data.  It also  
doesn't indicate what encryption technologies should be used by a File  
Writer to encrypt the file, what elements should/shouldn't be  
encrypted or how a File Reader would determine what type of decryption  
to perform.  Without including this information, the document is,  
essentially, suggesting (or nearly mandating) the implementation of  
non-interoperable security mechanisms.
ISSUE #2:  Option Scoping
There are options defined that are meant to apply to "the the whole  
IPFIX Transport Session (i.e., the IPFIX File in the common case)",  
such as the Export Session Details Option.  However, the document also  
indicates that the IPFIX file format is intended to support the free  
concatenation and splitting of IPFIX files at IPFIX Message  
boundaries.  Is it expected that session-scoped options will appear in  
all of the messages to which they apply?  If not, how would the  
session boundaries be determined?  I see that there are min and max  
export times in the session scoped options, but I don't see anything  
that prohibits (or even warns against) concatenating IPFIX sessions  
with overlapping times.  There is some discussion of this topic in  
section 7.1, where it is stated that File Readers may treat multiple  
files as a single Transport Session, but that they will not treat a  
single file as representing multiple Transport Sessions.  However,  
that doesn't seem to be consistent with the "free manipulability of  
IPFIX Files through concatenation, appending, and splitting (on IPFIX  
Message boundaries)" described in section 3.
ISSUE #3:  Data Compression
This issues is similar to the security issue described in ISSUE #1  
above.  The document indicates that data compression is desirable.   
Section 9.1 talks in details about issues with data compression and  
suggests that block compression be used.  However, the document does  
not mandate the implementation of any specific compression algorithm,  
nor does it indicate how a File Reader would determine that a file had  
been compressed and/or what type of compression was used.  Without  
this information, the document is suggesting the implementation of a  
non-interoperable compression mechanism.
IMO, the three issues I have listed above should be resolved before  
this document is published.
I have very recently started following the work of the ipfix WG, and I  
have recently subscribed to the mailing list.  So, I should be able to  
see replies that are sent to the list.
Thanks for your consideration of these issues.
Margaret

On Apr 14, 2009, at 6:32 AM, Romascanu, Dan (Dan) wrote:

>
>
> A URL of this Internet-Draft is:
> http://www.ietf.org/internet-drafts/draft-ietf-ipfix-file-03.txt
>
> Technical Summary
>
> This document describes a file format for the storage of flow data  
> based
> upon the IPFIX Protocol.  It proposes a set of requirements for
> flat-file, binary flow data file formats, then specifies the IPFIX  
> File
> format to meet these requirements based upon IPFIX Messages.
> This IPFIX File format is designed to facilitate interoperability and
> reusability among a wide variety of flow storage, processing, and
> analysis tools.
>
> Working Group Summary
>
> Work started on this document in 2006, and has gone through several
> revisions in response to mailing list discussion.  Originally it
> included the material about 'exporting type information,' but the WG
> split it out into a seperate RFC to make it more easily accessible for
> other uses.  The WG has reached a clear consensus on this draft.
>
> Document Quality
>
> This document has been reviwed by Benoit Claise, Paul Aitken, Andrew
> Johnson, and Gerhard Muenz, as well as by the WG chairs.  Vijay K.
> Gurbani reviewed the document on behaf of GenART. It specifies a file
> format for IPFIX data, making the point that this is simply another  
> data
> transport for IPFIX.  It does not make any changes to the IPFIX
> protocol.
>
> Personnel
>
> Nevil Brownlee is the Protocol Shepherd, Dan Romascanu is the
> shepherding AD.
>
> RFC Editor Note
>
> (Insert RFC Editor Note here or remove section)
>
> IRTF Note
>
> (Insert IRTF Note here or remove section)
>
> IESG Note
>
> (Insert IESG Note here or remove section)
>
> IANA Note
>
> (Insert IANA Note here or remove section)
>
>
>
>
>
>
> _______________________________________________
> OPS-DIR mailing list
> [email-masked]
> https://www.ietf.org/mailman/listinfo/ops-dir

---

From: "Romascanu, Dan (Dan)" <[email-masked]>
Date: Mon, 20 Apr 2009 21:42:47 +0200

Editors and Chairs,

I did not see yet any answer to the issues raised by Margaret in the
OPS-DIR review. Please address these before the IESG telechat on
Thursday. 

Dan

> -----Original Message-----
> From: Margaret Wasserman [mailto:[email-masked]] 
> Sent: Tuesday, April 14, 2009 6:08 PM
> To: Romascanu, Dan (Dan)
> Cc: [email-masked]; [email-masked]; IESG IESG
> Subject: Re: [OPS-DIR] FW: Evaluation: 
> draft-ietf-ipfix-file-03.txt to Proposed Standard
> 
> 
> I have some significant concerns about 
> draft-ietf-ipfix-file-03.txt that I would like to discuss.
> 
> This document does not seem to be consistent with its own 
> stated security requirements, nor does it specify any 
> interoperable security mechanisms.  In particular, sections 
> 5.6 of the document says:
> "5.6. Creator Authentication and Confidentiality "Archival 
> storage of flow data may also require assurance that no 
> unauthorized entity can read or modify the stored data. 
> Asymmetric- key cryptography can be applied to this problem, 
> by signing flow data with the private key of the creator, and 
> encrypting it with the public keys of those authorized to 
> read it. The ideal flow storage format will support the 
> encryption and signing of flow data. As with error 
> correction, this problem has been addressed well at a layer 
> below that addressed by this document. Instead of specifying 
> a particular choice of encryption technology, we can leverage 
> the fact Trammell, et al.  
> Expires April 27, 2009 [Page 12]>  Internet-Draft IPFIX Files October
20
> 08 that existing cryptographic technologies work quite well 
> on data stored in files to meet this requirement. Beyond 
> support for the use of TLS for transport over TCP or DTLS for 
> transport over SCTP or UDP, both of which provide transient 
> authentication and confidentiality, the IPFIX protocol does 
> not support this requirement directly. It is assumed that 
> this requirement will be met externally."
> I agree with this section, but I cannot find any place in the 
> document where it describes what type of file signing 
> technology should be implemented in File Readers and Writers, 
> what the certificates would look like and/or how they would 
> be bound to the file data.  It also doesn't indicate what 
> encryption technologies should be used by a File Writer to 
> encrypt the file, what elements should/shouldn't be encrypted 
> or how a File Reader would determine what type of decryption 
> to perform.  Without including this information, the document 
> is, essentially, suggesting (or nearly mandating) the 
> implementation of non-interoperable security mechanisms.
> ISSUE #2:  Option Scoping
> There are options defined that are meant to apply to "the the 
> whole IPFIX Transport Session (i.e., the IPFIX File in the 
> common case)", such as the Export Session Details Option.  
> However, the document also indicates that the IPFIX file 
> format is intended to support the free concatenation and 
> splitting of IPFIX files at IPFIX Message boundaries.  Is it 
> expected that session-scoped options will appear in all of 
> the messages to which they apply?  If not, how would the 
> session boundaries be determined?  I see that there are min 
> and max export times in the session scoped options, but I 
> don't see anything that prohibits (or even warns against) 
> concatenating IPFIX sessions with overlapping times.  There 
> is some discussion of this topic in section 7.1, where it is 
> stated that File Readers may treat multiple files as a single 
> Transport Session, but that they will not treat a single file 
> as representing multiple Transport Sessions.  However, that 
> doesn't seem to be consistent with the "free manipulability 
> of IPFIX Files through concatenation, appending, and 
> splitting (on IPFIX Message boundaries)" described in section 3.
> Section 9.1 talks in details about issues with data 
> compression and suggests that block compression be used.  
> However, the document does not mandate the implementation of 
> any specific compression algorithm, nor does it indicate how 
> a File Reader would determine that a file had been compressed 
> and/or what type of compression was used.  Without this 
> information, the document is suggesting the implementation of 
> a non-interoperable compression mechanism.
> IMO, the three issues I have listed above should be resolved 
> before this document is published.
> I have very recently started following the work of the ipfix 
> WG, and I have recently subscribed to the mailing list.  So, 
> I should be able to see replies that are sent to the list.
> > based upon the IPFIX Protocol.  It proposes a set of 
> requirements for 
> > File format to meet these requirements based upon IPFIX Messages.
> > data transport for IPFIX.  It does not make any changes to 
> the IPFIX 
> > protocol.
> >
> > Personnel
> >

---

From: Brian Trammell <[email-masked]>
Date: Tue, 21 Apr 2009 19:10:14 +0200

Margaret,

Thank you for your comprehensive review of the draft; my responses and  
comments appear inline below:

On Apr 14, 2009, at 5:18 PM, Margaret Wasserman wrote:

> security requirements, nor does it specify any interoperable  
> security mechanisms.  In particular, sections 5.6 of the document  
> says:
> key cryptography can be applied to this problem, by signing flow  
> data with the private key of the creator, and encrypting it with the  
> public keys of those authorized to read it. The ideal flow storage  
> format will support the encryption and signing of flow data. As with  
> error correction, this problem has been addressed well at a layer  
> below that addressed by this document. Instead of specifying a  
> particular choice of encryption technology, we can leverage the fact  
> that existing cryptographic technologies work quite well on data  
> of TLS for transport over TCP or DTLS for transport over SCTP or  
> UDP, both of which provide transient authentication and  
> confidentiality, the IPFIX protocol does not support this  
> requirement directly. It is assumed that this requirement will be  
> met externally."
> document where it describes what type of file signing technology  
> should be implemented in File Readers and Writers, what the  
> certificates would look like and/or how they would be bound to the  
> file data.  It also doesn't indicate what encryption technologies  
> should be used by a File Writer to encrypt the file, what elements  
> should/shouldn't be encrypted or how a File Reader would determine  
> what type of decryption to perform.  Without including this  
> information, the document is, essentially, suggesting (or nearly  
> mandating) the implementation of non-interoperable security  
> mechanisms.

This was something of a deliberate choice that is, indeed, probably  
better reviewed. Originally, we were thinking that compression and  
encryption were far more likely to be used for archival storage  
applications, where integration into _existing_ storage processes was  
more important than interoperability. The File Reader is assumed to be  
told how to decrypt data out of band. However, you are right in  
pointing out that without guidance this has serious interoperability  
concerns. Everything that you say here applies, as you say below, to  
compression as well (on which specifics below).

I'm not a crypto expert, so we will want some assistance from someone  
in the Security Area to at least verify the correctness and  
feasibility of our choice. The principle of least surprise would seem  
to dictate that we select something already standardized for  
encrypting files or file-like data. Symmetric and asymmetric crypto  
should both be supported. We would also like to actually be able to  
test interoperability, so working available implementations would be  
nice too.

On an initial admittedly cursory survey, CMS (RFCs 3852 and 4853,  
perhaps optionally with 5083) would seem to meet these requirements;  
we would need to provide recommendations on which content types are  
appropriate for which situations. This has the added advantage of  
direct compatibility with S/MIME, for people who want to mail IPFIX  
files about.

It seems structurally the best place to specify this would be a  
mention in 7.1 and 7.2, with further elaboration (including  
appropriate parts of current section 9.2) in a restructured section 9.

(This point consequently implies we'll need a MIME type, which we  
should probably have anyway.)

> such as the Export Session Details Option.  However, the document  
> also indicates that the IPFIX file format is intended to support the  
> free concatenation and splitting of IPFIX files at IPFIX Message  
> boundaries.  Is it expected that session-scoped options will appear  
> in all of the messages to which they apply?  If not, how would the  
> IPFIX Files through concatenation, appending, and splitting (on  
> IPFIX Message boundaries)" described in section 3.

Yes. This is indeed inconsistent, and we'd missed it.

Session boundaries are taken to be file boundaries, except in certain  
specific cases (large collections of very large files, where file  
sizes are limited by the underlying filesystem or other deployment- 
time concerns); the point of the sessionScope Information Element is  
to serve as a "null scope" allowing an Option to be bound to a Session  
(Options and Templates are taken to expire at Session end anyway, and  
are furthermore also implicitly scoped to Observation Domain).

Free manipulability is only available on files which don't contain any  
session-scoped information (and are furthermore not compressed or  
encrypted, of course); these are of course completely legal IPFIX  
Files. However, on further review, completely arbitrary concatenation  
operations seem to suffer from further minor problems (template and  
observation domain ID conflicts, sequence numbering) which the  
document must also address. My proposed remedy would be as follows:

1. Removal of any mention of free concatenation, appending, and  
splitting as a feature from the document. This was never a primary  
requirement of the file format; though it has often been discussed  
among the authors and the WG as a side-benefit of the format (over,  
e.g., file formats with explicit header and footer information), the  
text in section 3 seems to be the only place such language remains.

2. A short mention at an appropriate point in section 7.2 or 8 that  
when combining multiple files into one, or splitting files into parts,  
session-level information must be kept consistent, with an enumeration  
of said information.

> not mandate the implementation of any specific compression  
> algorithm, nor does it indicate how a File Reader would determine  
> that a file had been compressed and/or what type of compression was  
> used.  Without this information, the document is suggesting the  
> implementation of a non-interoperable compression mechanism.

As above, yes.

As for selecting a compression format, Section 9.1 states some  
desirable properties of an IPFIX file compression algorithm. bzip2  
meets these (it is indeed mentioned as an example in 9.1), but has the  
disadvantage of relatively slow compression, and an unclear reference  
to standardize against (RFC4880 references the homepage, for example).  
gzip (RFC1952) is more standards-friendly and in the general case  
faster to compress, but may have resynchronization issues in the face  
of errors.

It seems relatively easy to specify both (bz2 and gz streams can be  
distinguished by their magic numbers), but I'm not certain which I  
would recommend as mandatory.

I'll say what I said here for encryption: it seems structurally the  
best place to specify this would be a mention in 7.1 and 7.2, with  
further elaboration (including appropriate parts of current section  
9.1) in a restructured section 9.

> this document is published.

Agreed on all three points; hopefully these proposals will move us  
toward their resolution.

Thanks again, and best regards,

Brian

---

From: "Romascanu, Dan (Dan)" <[email-masked]>
Date: Tue, 21 Apr 2009 19:45:24 +0200

Thanks, Brian.

I am entering Margaret's issues in a COMMENT, waiting for one of the
AD's who did not yet enter a position to take these in a DISCUSS. 

Also we can probably use the advice of one of the Security AD's on issue
#1.

Dan

> From: Brian Trammell [mailto:[email-masked]] 
> Sent: Tuesday, April 21, 2009 8:10 PM
> To: Margaret Wasserman
> Cc: Romascanu, Dan (Dan); [email-masked]; IESG IESG; IETF 
> IPFIX Working Group; [email-masked]
> Subject: Re: [IPFIX] [OPS-DIR] FW: Evaluation: 
> Thank you for your comprehensive review of the draft; my 
> responses and comments appear inline below:
> 
> storage of 
> > flow data may also require assurance that no unauthorized 
> entity can 
> > read or modify the stored data. Asymmetric- key cryptography can be 
> > applied to this problem, by signing flow data with the 
> private key of 
> > the creator, and encrypting it with the public keys of those 
> > authorized to read it. The ideal flow storage format will 
> support the 
> > encryption and signing of flow data. As with error correction, this 
> > problem has been addressed well at a layer below that addressed by 
> > this document. Instead of specifying a particular choice of 
> encryption 
> > technology, we can leverage the fact that existing cryptographic 
> > technologies work quite well on data stored in files to meet this 
> > requirement. Beyond support for the use of TLS for 
> transport over TCP 
> > or DTLS for transport over SCTP or UDP, both of which provide 
> > transient authentication and confidentiality, the IPFIX 
> protocol does 
> > not support this requirement directly. It is assumed that this 
> > requirement will be met externally."
> > essentially, suggesting (or nearly
> This was something of a deliberate choice that is, indeed, 
> probably better reviewed. Originally, we were thinking that 
> compression and encryption were far more likely to be used 
> for archival storage applications, where integration into 
> _existing_ storage processes was more important than 
> interoperability. The File Reader is assumed to be told how 
> to decrypt data out of band. However, you are right in 
> pointing out that without guidance this has serious 
> interoperability concerns. Everything that you say here 
> applies, as you say below, to compression as well (on which 
> specifics below).
> 
> I'm not a crypto expert, so we will want some assistance from 
> someone in the Security Area to at least verify the 
> correctness and feasibility of our choice. The principle of 
> least surprise would seem to dictate that we select something 
> already standardized for encrypting files or file-like data. 
> Symmetric and asymmetric crypto should both be supported. We 
> would also like to actually be able to test interoperability, 
> so working available implementations would be nice too.
> 
> On an initial admittedly cursory survey, CMS (RFCs 3852 and 
> 4853, perhaps optionally with 5083) would seem to meet these 
> requirements; we would need to provide recommendations on 
> which content types are appropriate for which situations. 
> This has the added advantage of direct compatibility with 
> S/MIME, for people who want to mail IPFIX files about.
> 
> It seems structurally the best place to specify this would be 
> a mention in 7.1 and 7.2, with further elaboration (including 
> (This point consequently implies we'll need a MIME type, 
> which we should probably have anyway.)
> 
> > ISSUE #2:  Option Scoping
> Session boundaries are taken to be file boundaries, except in 
> certain specific cases (large collections of very large 
> files, where file sizes are limited by the underlying 
> filesystem or other deployment- time concerns); the point of 
> the sessionScope Information Element is to serve as a "null 
> scope" allowing an Option to be bound to a Session (Options 
> and Templates are taken to expire at Session end anyway, and 
> Free manipulability is only available on files which don't 
> contain any session-scoped information (and are furthermore 
> not compressed or encrypted, of course); these are of course 
> completely legal IPFIX Files. However, on further review, 
> completely arbitrary concatenation operations seem to suffer 
> from further minor problems (template and observation domain 
> ID conflicts, sequence numbering) which the document must 
> also address. My proposed remedy would be as follows:
> 
> 1. Removal of any mention of free concatenation, appending, 
> and splitting as a feature from the document. This was never 
> a primary requirement of the file format; though it has often 
> been discussed among the authors and the WG as a side-benefit 
> of the format (over, e.g., file formats with explicit header 
> and footer information), the text in section 3 seems to be 
> the only place such language remains.
> 
> 2. A short mention at an appropriate point in section 7.2 or 
> 8 that when combining multiple files into one, or splitting 
> files into parts, session-level information must be kept 
> consistent, with an enumeration of said information.
> 
> algorithm, 
> As for selecting a compression format, Section 9.1 states 
> some desirable properties of an IPFIX file compression 
> algorithm. bzip2 meets these (it is indeed mentioned as an 
> example in 9.1), but has the disadvantage of relatively slow 
> compression, and an unclear reference to standardize against 
> (RFC4880 references the homepage, for example).  
> gzip (RFC1952) is more standards-friendly and in the general 
> case faster to compress, but may have resynchronization 
> issues in the face of errors.
> 
> It seems relatively easy to specify both (bz2 and gz streams 
> can be distinguished by their magic numbers), but I'm not 
> certain which I would recommend as mandatory.
> 
> I'll say what I said here for encryption: it seems 
> structurally the best place to specify this would be a 
> appropriate parts of current section
> Agreed on all three points; hopefully these proposals will 
> move us toward their resolution.
>

---

From: Margaret Wasserman <[email-masked]>
Date: Thu, 23 Apr 2009 12:14:56 -0400

Hi Brian,

This all sounds good.  Please let me know if there is anything I can do to help.

Margaret

Brian Trammell wrote:
>> of encryption technology, we can leverage the fact that existing 
>> cryptographic technologies work quite well on data stored in files to 
>> meet this requirement. Beyond support for the use of TLS for transport 
>> over TCP or DTLS for transport over SCTP or UDP, both of which provide 
>> transient authentication and confidentiality, the IPFIX protocol does 
> told how to decrypt data out of band. However, you are right in pointing 
> out that without guidance this has serious interoperability concerns. 
> Everything that you say here applies, as you say below, to compression 
> as well (on which specifics below).
> 
> I'm not a crypto expert, so we will want some assistance from someone in 
> the Security Area to at least verify the correctness and feasibility of 
> our choice. The principle of least surprise would seem to dictate that 
> we select something already standardized for encrypting files or 
> file-like data. Symmetric and asymmetric crypto should both be 
> supported. We would also like to actually be able to test 
> interoperability, so working available implementations would be nice too.
> 
> perhaps optionally with 5083) would seem to meet these requirements; we 
> would need to provide recommendations on which content types are 
> appropriate for which situations. This has the added advantage of direct 
> compatibility with S/MIME, for people who want to mail IPFIX files about.
> 
> It seems structurally the best place to specify this would be a mention 
> in 7.1 and 7.2, with further elaboration (including appropriate parts of 
> current section 9.2) in a restructured section 9.
> 
> (This point consequently implies we'll need a MIME type, which we should 
> probably have anyway.)
> 
>> ISSUE #2:  Option Scoping
> specific cases (large collections of very large files, where file sizes 
> are limited by the underlying filesystem or other deployment-time 
> concerns); the point of the sessionScope Information Element is to serve 
> as a "null scope" allowing an Option to be bound to a Session (Options 
> and Templates are taken to expire at Session end anyway, and are 
> furthermore also implicitly scoped to Observation Domain).
> 
> encrypted, of course); these are of course completely legal IPFIX Files. 
> However, on further review, completely arbitrary concatenation 
> observation domain ID conflicts, sequence numbering) which the document 
> must also address. My proposed remedy would be as follows:
> 
> requirement of the file format; though it has often been discussed among 
> the authors and the WG as a side-benefit of the format (over, e.g., file 
> formats with explicit header and footer information), the text in 
> section 3 seems to be the only place such language remains.
> 
> 2. A short mention at an appropriate point in section 7.2 or 8 that when 
> combining multiple files into one, or splitting files into parts, 
> As for selecting a compression format, Section 9.1 states some desirable 
> properties of an IPFIX file compression algorithm. bzip2 meets these (it 
> is indeed mentioned as an example in 9.1), but has the disadvantage of 
> relatively slow compression, and an unclear reference to standardize 
> against (RFC4880 references the homepage, for example). gzip (RFC1952) 
> is more standards-friendly and in the general case faster to compress, 
> but may have resynchronization issues in the face of errors.
> 
> distinguished by their magic numbers), but I'm not certain which I would 
> recommend as mandatory.
> 
> I'll say what I said here for encryption: it seems structurally the best 
> place to specify this would be a mention in 7.1 and 7.2, with further 
> elaboration (including appropriate parts of current section 9.1) in a 
> restructured section 9.
>