Sinodun IT

Magdalen Centre Oxford Science Park Oxford OX4 4GA United Kingdom jad@sinodun.com

Sinodun IT

Magdalen Centre Oxford Science Park Oxford OX4 4GA United Kingdom jim@sinodun.com

Sinodun IT

Magdalen Centre Oxford Science Park Oxford OX4 4GA United Kingdom sara@sinodun.com

ICANN

12025 Waterfront Drive Suite 300 Los Angeles CA 90094-2536 terry.manderson@icann.org

ICANN

12025 Waterfront Drive Suite 300 Los Angeles CA 90094-2536 john.bond@icann.org

Operations Area dnsop DNS This document describes a data representation for collections of DNS messages. The format is designed for efficient storage and transmission of large packet captures of DNS traffic; it attempts to minimize the size of such packet capture files but retain the full DNS message contents along with the most useful transport metadata. It is intended to assist with the development of DNS traffic monitoring applications.

There has long been a need for server operators to collect DNS queries and responses on authoritative and recursive name servers for monitoring and analysis. This data is used in a number of ways including traffic monitoring, analyzing network attacks and "day in the life" (DITL) analysis. A wide variety of tools already exist that facilitate the collection of DNS traffic data, such as DSC , packetq , dnscap and dnstap . However, there is no standard exchange format for large DNS packet captures. The PCAP or PCAP-NG formats are typically used in practice for packet captures, but these file formats can contain a great deal of additional information that is not directly pertinent to DNS traffic analysis and thus unnecessarily increases the capture file size. Additionally these tools and formats typically have no filter mechanism to selectively record only certain fields at capture time, requiring post-processing for anonymization or pseudonymization of data to protect user privacy. There has also been work on using text based formats to describe DNS packets such as , , but these are largely aimed at producing convenient representations of single messages. Many DNS operators may receive hundreds of thousands of queries per second on a single name server instance so a mechanism to minimize the storage and transmission size (and therefore upload overhead) of the data collected is highly desirable. The format described in this document, C-DNS (Compacted-DNS), focusses on the problem of capturing and storing large packet capture files of DNS traffic with the following goals in mind: Minimize the file size for storage and transmission. Minimize the overhead of producing the packet capture file and the cost of any further (general purpose) compression of the file. This document contains: A discussion of some common use cases in which DNS data is collected, see . A discussion of the major design considerations in developing an efficient data representation for collections of DNS messages, see . A description of why CBOR was chosen for this format, see . A conceptual overview of the C-DNS format, see . The definition of the C-DNS format for the collection of DNS messages, see . Notes on converting C-DNS data to PCAP format, see . Some high level implementation considerations for applications designed to produce C-DNS, see .

The key words "MUST", "MUST NOT", "REQUIRED", "SHALL", "SHALL NOT", "SHOULD", "SHOULD NOT", "RECOMMENDED", "NOT RECOMMENDED", "MAY", and "OPTIONAL" in this document are to be interpreted as described in BCP 14 when, and only when, they appear in all capitals, as shown here. "Packet" refers to an individual IPv4 or IPv6 packet. Typically packets are UDP datagrams, but may also be part of a TCP data stream. "Message", unless otherwise qualified, refers to a DNS payload extracted from a UDP datagram or a TCP data stream. The parts of DNS messages are named as they are in . Specifically, the DNS message has five sections: Header, Question, Answer, Authority, and Additional. Pairs of DNS messages are called a Query and a Response.

From a purely server operator perspective, collecting full packet captures of all packets going in or out of a name server provides the most comprehensive picture of network activity. However, there are several design choices or other limitations that are common to many DNS installations and operators. DNS servers are hosted in a variety of situations: Self-hosted servers Third party hosting (including multiple third parties) Third party hardware (including multiple third parties) Data is collected under different conditions: On well-provisioned servers running in a steady state On heavily loaded servers On virtualized servers On servers that are under DoS attack On servers that are unwitting intermediaries in DoS attacks Traffic can be collected via a variety of mechanisms: Within the name server implementation itself On the same hardware as the name server itself Using a network tap on an adjacent host to listen to DNS traffic Using port mirroring to listen from another host The capabilities of data collection (and upload) networks vary: Out-of-band networks with the same capacity as the in-band network Out-of-band networks with less capacity than the in-band network Everything being on the in-band network Thus, there is a wide range of use cases from very limited data collection environments (third party hardware, servers that are under attack, packet capture on the name server itself and no out-of-band network) to "limitless" environments (self hosted, well provisioned servers, using a network tap or port mirroring with an out-of-band networks with the same capacity as the in-band network). In the former, it is infeasible to reliably collect full packet captures, especially if the server is under attack. In the latter case, collection of full packet captures may be reasonable. As a result of these restrictions, the C-DNS data format is designed with the most limited use case in mind such that: data collection will occur on the same hardware as the name server itself collected data will be stored on the same hardware as the name server itself, at least temporarily collected data being returned to some central analysis system will use the same network interface as the DNS queries and responses there can be multiple third party servers involved Because of these considerations, a major factor in the design of the format is minimal storage size of the capture files. Another significant consideration for any application that records DNS traffic is that the running of the name server software and the transmission of DNS queries and responses are the most important jobs of a name server; capturing data is not. Any data collection system co-located with the name server needs to be intelligent enough to carefully manage its CPU, disk, memory and network utilization. This leads to designing a format that requires a relatively low overhead to produce and minimizes the requirement for further potentially costly compression. However, it is also essential that interoperability with less restricted infrastructure is maintained. In particular, it is highly desirable that the collection format should facilitate the re-creation of common formats (such as PCAP) that are as close to the original as is realistic given the restrictions above.

This section presents some of the major design considerations used in the development of the C-DNS format. The basic unit of data is a combined DNS Query and the associated Response (a "Q/R data item"). The same structure will be used for unmatched Queries and Responses. Queries without Responses will be captured omitting the response data. Responses without queries will be captured omitting the Query data (but using the Question section from the response, if present, as an identifying QNAME). Rationale: A Query and Response represents the basic level of a client's interaction with the server. Also, combining the Query and Response into one item often reduces storage requirements due to commonality in the data of the two messages. In the context of generating a C-DNS file it is assumed that only those DNS payloads which can be parsed to produce a well-formed DNS message are stored in the C-DNS format and that all other messages will be (optionally) recorded as malformed messages. Parsing a well-formed message means as a minimum: The packet has a well-formed 12 byte DNS Header with a recognised OPCODE. The section counts are consistent with the section contents. All of the resource records can be fully parsed. All top level fields in each Q/R data item will be optional. Rationale: Different operators will have different requirements for data to be available for analysis. Operators with minimal requirements should not have to pay the cost of recording full data, though this will limit the ability to perform certain kinds of data analysis and also to reconstruct packet captures. For example, omitting the resource records from a Response will reduce the C-DNS file size; in principle responses can be synthesized if there is enough context. Operators may have different policies for collecting user data and can choose to omit or anonymize certain fields at capture time e.g. client address. Multiple Q/R data items will be collected into blocks in the format. Common data in a block will be abstracted and referenced from individual Q/R data items by indexing. The maximum number of Q/R data items in a block will be configurable. Rationale: This blocking and indexing provides a significant reduction in the volume of file data generated. Although this introduces complexity, it provides compression of the data that makes use of knowledge of the DNS message structure. It is anticipated that the files produced can be subject to further compression using general purpose compression tools. Measurements show that blocking significantly reduces the CPU required to perform such strong compression. See . Examples of commonality between DNS messages are that in most cases the QUESTION RR is the same in the query and response, and that there is a finite set of query signatures (based on a subset of attributes). For many authoritative servers there is very likely to be a finite set of responses that are generated, of which a large number are NXDOMAIN. Traffic metadata can optionally be included in each block. Specifically, counts of some types of non-DNS packets (e.g. ICMP, TCP resets) sent to the server may be of interest. The wire format content of malformed DNS messages may optionally be recorded. Rationale: Any structured capture format that does not capture the DNS payload byte for byte will be limited to some extent in that it cannot represent malformed DNS messages. Only those messages that can be fully parsed and transformed into the structured format can be fully represented. Note, however, this can result in rather misleading statistics. For example, a malformed query which cannot be represented in the C-DNS format will lead to the (well formed) DNS responses with error code FORMERR appearing as 'unmatched'. Therefore it can greatly aid downstream analysis to have the wire format of the malformed DNS messages available directly in the C-DNS file.

This document presents a detailed format description using CBOR, the Concise Binary Object Representation defined in . The choice of CBOR was made taking a number of factors into account. CBOR is a binary representation, and thus is economical in storage space. Other binary representations were investigated, and whilst all had attractive features, none had a significant advantage over CBOR. See for some discussion of this. CBOR is an IETF specification and familiar to IETF participants. It is based on the now-common ideas of lists and objects, and thus requires very little familiarization for those in the wider industry. CBOR is a simple format, and can easily be implemented from scratch if necessary. More complex formats require library support which may present problems on unusual platforms. CBOR can also be easily converted to text formats such as JSON () for debugging and other human inspection requirements. CBOR data schemas can be described using CDDL .

The following figures show purely schematic representations of the C-DNS format to convey the high-level structure of the C-DNS format. provides a detailed discussion of the CBOR representation and individual elements. Figure 1 shows the C-DNS format at the top level including the file header and data blocks. The Query/Response data items, Address/Event Count data items and Malformed Message data items link to various Block tables.

+-------+ + C-DNS | +-------+--------------------------+ | File type identifier | +----------------------------------+ | File preamble | | +--------------------------------+ | | Format version info | | +--------------------------------+ | | Block parameters | +-+--------------------------------+ | Block | | +--------------------------------+ | | Block preamble | | +--------------------------------+ | | Block statistics | | +--------------------------------+ | | Block tables | | +--------------------------------+ | | Query/Response data items | | +--------------------------------+ | | Address/Event Count data items | | +--------------------------------+ | | Malformed Message data items | +-+--------------------------------+ | Block | | +--------------------------------+ | | Block preamble | | +--------------------------------+ | | Block statistics | | +--------------------------------+ | | Block tables | | +--------------------------------+ | | Query/Response data items | | +--------------------------------+ | | Address/Event Count data items | | +--------------------------------+ | | Malformed Message data items | +-+--------------------------------+ | Further Blocks... | +----------------------------------+

Figure 2 shows some more detailed relationships within each block, specifically those between the Query/Response data item and the relevant Block tables.

+----------------+ | Query/Response | +-------------------------+ | Time offset | +-------------------------+ +------------------+ | Client address |------------>| IP address array | +-------------------------+ +------------------+ | Client port | +-------------------------+ +------------------+ | Transaction ID | +------>| Name/RDATA array |<------+ +-------------------------+ | +------------------+ | | Query signature |--+ | | +-------------------------+ | | +-----------------+ | | Client hoplimit (q) | +--)------>| Query Signature | | +-------------------------+ | +-----------------+------+ | | Response delay (r) | | | Server address | | +-------------------------+ | +------------------------+ | | Query name |--+--+ | Server port | | +-------------------------+ | +------------------------+ | | Query size (q) | | | Transport flags | | +-------------------------+ | +------------------------+ | | Response size (r) | | | QR type | | +-------------------------+ | +------------------------+ | | Response processing (r) | | | QR signature flags | | | +-----------------------+ | +------------------------+ | | | Bailiwick index |--+ | Query OPCODE (q) | | | +-----------------------+ +------------------------+ | | | Flags | | QR DNS flags | | +-+-----------------------+ +------------------------+ | | Extra query info (q) | | Query RCODE (q) | | | +-----------------------+ +------------------------+ | | | Question |--+---+ +--+-Query Class/Type (q) | | | +-----------------------+ | | +------------------------+ | | | Answer |--+ | | | Query QD count (q) | | | +-----------------------+ | | | +------------------------+ | | | Authority |--+ | | | Query AN count (q) | | | +-----------------------+ | | | +------------------------+ | | | Additional |--+ | | | Query NS count (q) | | +-+-----------------------+ | | | +------------------------+ | | Extra response info (r) | |-+ | | | Query EDNS version (q) | | | +-----------------------+ | | | | +------------------------+ | | | Answer |--+ | | | | EDNS UDP size (q) | | | +-----------------------+ | | | | +------------------------+ | | | Authority |--+ | | | | Query Opt RDATA (q) | | | +-----------------------+ | | | | +------------------------+ | | | Additional |--+ | | | | Response RCODE (r) | | +-+-----------------------+ | | | +------------------------+ | | | | | | | | | + -----------------------------+ | +----------+ | | | | | | + -----------------------------+ | | | | +---------------+ +----------+ | | | +->| Question list |->| Question | | | | | array | | array | | | | +---------------+ +----------+--+ | | | | Name |--+------)------------------+ | +-------------+ | | +------------+ | | Class/type |--)---+--+->| Class/Type | | +-------------+ | | | array | | | | +------------+--+ | | | | Class | | +---------------+ +----------+ | | +---------------+ +--->| RR list array |->| RR array | | | | Type | +---------+-----+ +----------+--+ | | +---------------+ | Name |--+ | +-------------+ | | Class/type |------+ +-------------+

In Figure 2 data items annotated (q) are only present when a query/response has a query, and those annotated (r) are only present when a query/response response is present. A C-DNS file begins with a file header containing a File Type Identifier and a File Preamble. The File Preamble contains information on the file Format Version and an array of Block Parameters items (the contents of which include Collection and Storage Parameters used for one or more blocks). The file header is followed by a series of data Blocks. A Block consists of a Block Preamble item, some Block Statistics for the traffic stored within the Block and then various arrays of common data collectively called the Block Tables. This is then followed by an array of the Query/Response data items detailing the queries and responses stored within the Block. The array of Query/Response data items is in turn followed by the Address/Event Counts data items (an array of per-client counts of particular IP events) and then Malformed Message data items (an array of malformed messages that stored in the Block). The exact nature of the DNS data will affect what block size is the best fit, however sample data for a root server indicated that block sizes up to 10,000 Q/R data items give good results. See for more details. This design exploits data commonality and block based storage to minimise the C-DNS file size. As a result C-DNS cannot be streamed below the level of a block.

The details of the Block Parameters items are not shown in the diagrams but are discussed here for context. An array of Block Parameters items is stored in the File Preamble (with a minimum of one item at index 0); a Block Parameters item consists of a collection of Storage and Collection Parameters that applies to any given Block. An array is used in order to support use cases such as wanting to merge C-DNS files from different sources. The Block Preamble item then contains an optional index for the Block Parameters item that applies for that Block; if not present the index defaults to 0. Hence, in effect, a global Block Parameters item is defined which can then be overridden per Block.

The Block Parameters item includes a Storage Parameters item - this contains information about the specific data fields stored in the C-DNS file. These parameters include: The sub-second timing resolution used by the data. Information (hints) on which optional data are omitted. See . Recorded OPCODES and RR types . See . Flags indicating, for example, whether the data is sampled or anonymized. See and . Client and server IPv4 and IPv6 address prefixes. See

To enable implementations to store data to their precise requirements in as space-efficient manner as possible, all fields in the following arrays are optional: Query/Response Query Signature Malformed messages In other words, an implementation can choose to omit any data item that is not required for its use case. In addition, implementations may be configured to not record all RRs, or only record messages with certain OPCODES. This does, however, mean that a consumer of a C-DNS file faces two problems: How can it quickly determine if a file definitely does not contain the data items it requires to complete a particular task (e.g. reconstructing query traffic or performing a specific piece of data analysis)? How can it determine if a data item is not present because it was: explicitly not recorded or the data item was not available/present. For example, capturing C-DNS data from within a nameserver implementation makes it unlikely that the Client Hoplimit can be recorded. Or, if there is no query ARCount recorded and no query OPT RDATA recorded, is that because no query contained an OPT RR, or because that data was not stored? The Storage Parameters therefore also contains a Storage Hints item which specifies which items the encoder of the file omits from the stored data and will therefore never be present. (This approach is taken because a flag that indicated which items were included for collection would not guarantee that the item was present, only that it might be.) An implementation decoding that file can then use these to quickly determine whether the input data is rich enough for its needs.

Also included in the Storage Parameters are explicit arrays listing the RR types and the OPCODEs to be recorded. These remove any ambiguity over whether messages containing particular OPCODEs or are not present because they did not occur, or because the implementation is not configured to record them. In the case of OPCODEs, for a message to be fully parsable, the OPCODE must be known to the collecting implementation. Any message with an OPCODE unknown to the collecting implementation cannot be validated as correctly formed, and so must be treated as malformed. Messages with OPCODES known to the recording application but not listed in the Storage Parameters are discarded by the recording application during C-DNS capture (regardless of whether they are malformed or not). In the case of RR records, each record in a message must be fully parsable, including parsing the record RDATA, as otherwise the message cannot be validated as correctly formed. Any RR record with an RR type not known to the collecting implementation cannot be validated as correctly formed, and so must be treated as malformed. Once a message is correctly parsed, an implementation is free to record only a subset of the RR records present.

The Storage Parameters contains flags that can be used to indicate if: the data is anonymized, the data is produced from sample data, or names in the data have been normalized (converted to uniform case). The Storage Parameters also contains optional fields holding details of the sampling method used and the anonymization method used. It is RECOMMENDED these fields contain URIs pointing to resources describing the methods used. See for further discussion of anonymization and normalization.

The format can store either full IP addresses or just IP prefixes, the Storage Parameters contains fields to indicate if only IP prefixes were stored. If the IP address prefixes are absent, then full addresses are stored. In this case the IP version can be directly inferred from the stored address length and the fields qr-transport-flags in QueryResponseSignature and mm-transport-flags in MalformedMessageData (which contain the IP version bit) are optional. If IP address prefixes are given, only the prefix bits of addresses are stored. In this case the fields qr-transport-flags in QueryResponseSignature and mm-transport-flags in MalformedMessageData MUST be present, so that the IP version can be determined. See and . As an example of storing only IP prefixes, if a client IPv6 prefix of 48 is specified, a client address of 2001:db8:85a3::8a2e:370:7334 will be stored as 0x20010db885a3, reducing address storage space requirements. Similarly, if a client IPv4 prefix of 16 is specified, a client address of 192.0.2.1 will be stored as 0xc000 (192.0).

The CDDL definition for the C-DNS format is given in .

All map keys are integers with values specified in the CDDL. String keys would significantly bloat the file size. All key values specified are positive integers under 24, so their CBOR representation is a single byte. Positive integer values not currently used as keys in a map are reserved for use in future standard extensions. Implementations may choose to add additional implementation-specific entries to any map. Negative integer map keys are reserved for these values. Key values from -1 to -24 also have a single byte CBOR representation, so such implementation-specific extensions are not at any space efficiency disadvantage. An item described as an index is the index of the data item in the referenced array. Indexes are 0-based.

The following sections present the C-DNS specification in tabular format with a detailed description of each item. In all quantities that contain bit flags, bit 0 indicates the least significant bit, i.e. flag n in quantity q is on if (q & (1 << n)) != 0. For the sake of readability, all type and field names defined in the CDDL definition are shown in double quotes. Type names are by convention camel case (e.g. BlockTable), field names are lower-case with hyphens (e.g. block-tables). For the sake of brevity, the following conventions are used in the tables: The column M marks whether items in a map are mandatory. X - Mandatory items. C - Conditionally mandatory item. Such items are usually optional but may be mandatory in some configurations. If the column is empty, the item is optional. The column T gives the CBOR data type of the item. U - Unsigned integer I - Signed integer (i.e. CBOR unsigned or negative integer) B - Boolean S - Byte string T - Text string M - Map A - Array In the case of maps and arrays, more information on the type of each value, include the CDDL definition name if applicable, is given in the description.

A C-DNS file has an outer structure File, a map that contains the following: Field M T Description file-type-idXTString "C-DNS" identifying the file type. file-preambleXMVersion and parameter information for the whole file. Map of type FilePreamble, see . file-blocksXAArray of items of type Block, see . The array may be empty if the file contains no data.

Information about data in the file. A map containing the following: Field M T Description major-format-versionXUUnsigned integer '1'. The major version of format used in file. See . minor-format-versionXUUnsigned integer '0'. The minor version of format used in file. See . private-versionUVersion indicator available for private use by implementations. block-parametersXAArray of items of type BlockParameters, see . The array must contain at least one entry. (The block-parameters-index item in each BlockPreamble indicates which array entry applies to that Block.)

Parameters relating to data storage and collection which apply to one or more items of type Block. A map containing the following: Field M T Description storage-parametersXMParameters relating to data storage in a Block item. Map of type StorageParameters, see . collection-parametersMParameters relating to collection of the data in a Block item. Map of type CollectionParameters, see .

Parameters relating to how data is stored in the items of type Block. A map containing the following: Field M T Description ticks-per-secondXUSub-second timing is recorded in ticks. This specifies the number of ticks in a second. max-block-itemsXUThe maximum number of items stored in any of the arrays in a Block item (Q/R items, address event counts or malformed messages). An indication to a decoder of the resources needed to process the file. storage-hintsXMCollection of hints as to which fields are omitted in the arrays that have optional fields. Map of type StorageHints, see . opcodesXAArray of OPCODES (unsigned integers, each in the range 0 to 15 inclusive) recorded by the collection implementation. See . rr-typesXAArray of RR types (unsigned integers, each in the range 0 to 65535 inclusive) recorded by the collection implementation. See . storage-flagsUBit flags indicating attributes of stored data. Bit 0. 1 if the data has been anonymized. Bit 1. 1 if the data is sampled data. Bit 2. 1 if the names have been normalized (converted to uniform case). client-address -prefix-ipv4UIPv4 client address prefix length, in the range 1 to 32 inclusive. If specified, only the address prefix bits are stored. client-address -prefix-ipv6UIPv6 client address prefix length, in the range 1 to 128 inclusive. If specified, only the address prefix bits are stored. server-address -prefix-ipv4UIPv4 server address prefix length, in the range 1 to 32 inclusive. If specified, only the address prefix bits are stored. server-address -prefix-ipv6UIPv6 server address prefix length, in the range 1 to 128 inclusive. If specified, only the address prefix bits are stored. sampling-methodTInformation on the sampling method used. See . anonymization -methodTInformation on the anonymization method used. See .

An indicator of which fields the collecting implementation omits in the maps with optional fields. A map containing the following: Field M T Description query-response -hintsXUHints indicating which QueryResponse fields are candidates for capture or omitted, see section . If a bit is unset, the field is omitted from the capture. Bit 0. time-offset Bit 1. client-address-index Bit 2. client-port Bit 3. transaction-id Bit 4. qr-signature-index Bit 5. client-hoplimit Bit 6. response-delay Bit 7. query-name-index Bit 8. query-size Bit 9. response-size Bit 10. response-processing-data Bit 11. query-question-sections Bit 12. query-answer-sections Bit 13. query-authority-sections Bit 14. query-additional-sections Bit 15. response-answer-sections Bit 16. response-authority-sections Bit 17. response-additional-sections query-response -signature-hintsXUHints indicating which QueryResponseSignature fields are candidates for capture or omitted, see section . If a bit is unset, the field is omitted from the capture. Bit 0. server-address Bit 1. server-port Bit 2. qr-transport-flags Bit 3. qr-type Bit 4. qr-sig-flags Bit 5. query-opcode Bit 6. dns-flags Bit 7. query-rcode Bit 8. query-class-type Bit 9. query-qdcount Bit 10. query-ancount Bit 11. query-nscount Bit 12. query-arcount Bit 13. query-edns-version Bit 14. query-udp-size Bit 15. query-opt-rdata Bit 16. response-rcode rr-hintsXUHints indicating which optional RR fields are candidates for capture or omitted, see . If a bit is unset, the field is omitted from the capture. Bit 0. ttl Bit 1. rdata-index other-data-hintsXUHints indicating which other data types are omitted. If a bit is unset, the the data type is omitted from the capture. Bit 0. malformed-messages Bit 1. address-event-counts

Parameters providing information to how data in the file was collected (applicable for some, but not all collection environments). The values are informational only and serve as hints to downstream analysers as to the configuration of a collecting implementation. They can provide context when interpreting what data is present/absent from the capture but cannot necessarily be validated against the data captured. These parameters have no default. If they do not appear, nothing can be inferred about their value. A map containing the following items: Field M T Description query-timeoutUTo be matched with a query, a response must arrive within this number of seconds. skew-timeoutUThe network stack may report a response before the corresponding query. A response is not considered to be missing a query until after this many micro-seconds. snaplenUCollect up to this many bytes per packet. promiscBtrue if promiscuous mode was enabled on the interface, false otherwise. interfacesAArray of identifiers (of type text string) of the interfaces used for collection. server-addressesAArray of server collection IP addresses (of type byte string). Hint for downstream analysers; does not affect collection. vlan-idsAArray of identifiers (of type unsigned integer, each in the range 1 to 4094 inclusive) of VLANs selected for collection. VLAN IDs are unique only within an administrative domain. filterTtcpdump style filter for input. generator-idTImplementation specific human-readable string identifying the collection method. host-idTString identifying the collecting host. Empty if converting an existing packet capture file.

Container for data with common collection and storage parameters. A map containing the following: Field M T Description block-preambleXMOverall information for the Block item. Map of type BlockPreamble, see . block-statisticsMStatistics about the Block item. Map of type BlockStatistics, see . block-tablesMThe arrays containing data referenced by individual QueryResponse or MalformedMessage items. Map of type BlockTables, see . query-responsesADetails of individual DNS Q/R data items. Array of items of type QueryResponse, see . If present, the array must not be empty. address-event -countsAPer client counts of ICMP messages and TCP resets. Array of items of type AddressEventCount, see . If present, the array must not be empty. malformed-messagesADetails of malformed DNS messages. Array of items of type MalformedMessage, see . If present, the array must not be empty.

Overall information for a Block item. A map containing the following: Field M T Description earliest-timeCAA timestamp (2 unsigned integers, Timestamp) for the earliest record in the Block item. The first integer is the number of seconds since the POSIX epoch (time_t), excluding leap seconds. The second integer is the number of ticks (see ) since the start of the second. This field is mandatory unless all block items containing a time offset from the start of the block also omit that time offset. block-parameters -indexUThe index of the item in the block-parameters array (in the file-premable item) applicable to this block. If not present, index 0 is used. See .

Basic statistical information about a Block item. A map containing the following: Field M T Description processed-messagesUTotal number of DNS messages processed from the input traffic stream during collection of data in this Block item. qr-data-itemsUTotal number of Q/R data items in this Block item. unmatched-queriesUNumber of unmatched queries in this Block item. unmatched-responsesUNumber of unmatched responses in this Block item. discarded-opcodeUNumber of DNS messages processed from the input traffic stream during collection of data in this Block item but not recorded because their OPCODE is not in the list to be collected. malformed-itemsUNumber of malformed messages found in input for this Block item.

Map of arrays containing data referenced by individual QueryResponse or MalformedMessage items in this Block. Each element is an array which, if present, must not be empty. An item in the qlist array contains indexes to values in the qrr array. Therefore, if qlist is present, qrr must also be present. Similarly, if rrlist is present, rr must also be present. The map contains the following items: Field M T Description ip-addressAArray of IP addresses, in network byte order (of type byte string). If client or server address prefixes are set, only the address prefix bits are stored. Each string is therefore up to 4 bytes long for an IPv4 address, or up to 16 bytes long for an IPv6 address. See . classtypeAArray of RR class and type information. Type is ClassType, see . name-rdataAArray where each entry is the contents of a single NAME or RDATA in wire format (of type byte string). Note that NAMEs, and labels within RDATA contents, are full domain names or labels; no name compression is used on the individual names/labels within the format. qr-sigAArray Q/R data item signatures. Type is QueryResponseSignature, see . qlistAArray of type QuestionList. A QuestionList is an array of unsigned integers, indexes to Question items in the qrr array. qrrAArray of type Question. Each entry is the contents of a single question, where a question is the second or subsequent question in a query. See . rrlistAArray of type RRList. An RRList is an array of unsigned integers, indexes to RR items in the rr array. rrAArray of type RR. Each entry is the contents of a single RR. See . malformed-message -dataAArray of the contents of malformed messages. Array of type MalformedMessageData, see .

RR class and type information. A map containing the following: Field M T Description typeXUTYPE value . classXUCLASS value .

Elements of a Q/R data item that are often common between multiple individual Q/R data items. A map containing the following: Field M T Description server-address -indexUThe index in the item in the ip-address array of the server IP address. See . server-portUThe server port. qr-transport-flagsCUBit flags describing the transport used to service the query. Same definition as mm-transport-flags in , with an additional indicator for trailing bytes, see . Bit 0. IP version. 0 if IPv4, 1 if IPv6. See . Bit 1-4. Transport. 4 bit unsigned value where 0 = UDP, 1 = TCP, 2 = TLS, 3 = DTLS , 4 = DoH . Values 5-15 are reserved for future use. Bit 5. 1 if trailing bytes in query packet. See . qr-typeUType of Query/Response transaction. 0 = Stub. A query from a stub resolver. 1 = Client. An incoming query to a recursive resolver. 2 = Resolver. A query sent from a recursive resolver to an authorative resolver. 3 = Authorative. A query to an authorative resolver. 4 = Forwarder. A query sent from a recursive resolver to an upstream recursive resolver. 5 = Tool. A query sent to a server by a server tool. qr-sig-flagsUBit flags explicitly indicating attributes of the message pair represented by this Q/R data item (not all attributes may be recorded or deducible). Bit 0. 1 if a Query was present. Bit 1. 1 if a Response was present. Bit 2. 1 if a Query was present and it had an OPT Resource Record. Bit 3. 1 if a Response was present and it had an OPT Resource Record. Bit 4. 1 if a Query was present but had no Question. Bit 5. 1 if a Response was present but had no Question (only one query-name-index is stored per Q/R item). query-opcodeUQuery OPCODE. qr-dns-flagsUBit flags with values from the Query and Response DNS flags. Flag values are 0 if the Query or Response is not present. Bit 0. Query Checking Disabled (CD). Bit 1. Query Authenticated Data (AD). Bit 2. Query reserved (Z). Bit 3. Query Recursion Available (RA). Bit 4. Query Recursion Desired (RD). Bit 5. Query TrunCation (TC). Bit 6. Query Authoritative Answer (AA). Bit 7. Query DNSSEC answer OK (DO). Bit 8. Response Checking Disabled (CD). Bit 9. Response Authenticated Data (AD). Bit 10. Response reserved (Z). Bit 11. Response Recursion Available (RA). Bit 12. Response Recursion Desired (RD). Bit 13. Response TrunCation (TC). Bit 14. Response Authoritative Answer (AA). query-rcodeUQuery RCODE. If the Query contains OPT , this value incorporates any EXTENDED_RCODE_VALUE . query-classtype -indexUThe index to the item in the the classtype array of the CLASS and TYPE of the first Question. See . query-qd-countUThe QDCOUNT in the Query, or Response if no Query present. query-an-countUQuery ANCOUNT. query-ns-countUQuery NSCOUNT. query-ar-countUQuery ARCOUNT. edns-versionUThe Query EDNS version. udp-buf-sizeUThe Query EDNS sender's UDP payload size. opt-rdata-indexUThe index in the name-rdata array of the OPT RDATA. See . response-rcodeUResponse RCODE. If the Response contains OPT , this value incorporates any EXTENDED_RCODE_VALUE .

Details on individual Questions in a Question section. A map containing the following: Field M T Description name-indexXUThe index in the name-rdata array of the QNAME. See . classtype-indexXUThe index in the classtype array of the CLASS and TYPE of the Question. See .

Details on individual Resource Records in RR sections. A map containing the following: Field M T Description name-indexXUThe index in the name-rdata array of the NAME. See . classtype-indexXUThe index in the classtype array of the CLASS and TYPE of the RR. See . ttlUThe RR Time to Live. rdata-indexUThe index in the name-rdata array of the RR RDATA. See .

Details on malformed message items in this Block item. A map containing the following: Field M T Description server-address -indexUThe index in the ip-address array of the server IP address. See . server-portUThe server port. mm-transport-flagsCUBit flags describing the transport used to service the query, see . Bit 0. IP version. 0 if IPv4, 1 if IPv6 Bit 1-4. Transport. 4 bit unsigned value where 0 = UDP, 1 = TCP, 2 = TLS, 3 = DTLS , 4 = DoH . Values 5-15 are reserved for future use. mm-payloadSThe payload (raw bytes) of the DNS message.

Details on individual Q/R data items. Note that there is no requirement that the elements of the query-responses array are presented in strict chronological order. A map containing the following items: Field M T Description time-offsetUQ/R timestamp as an offset in ticks (see ) from earliest-time. The timestamp is the timestamp of the Query, or the Response if there is no Query. client-address-indexUThe index in the ip-address array of the client IP address. See . client-portUThe client port. transaction-idUDNS transaction identifier. qr-signature-indexUThe index in the qr-sig array of the QueryResponseSignature item. See . client-hoplimitUThe IPv4 TTL or IPv6 Hoplimit from the Query packet. response-delayIThe time difference between Query and Response, in ticks (see ). Only present if there is a query and a response. The delay can be negative if the network stack/capture library returns packets out of order. query-name-indexUThe index in the name-rdata array of the item containing the QNAME for the first Question. See . query-sizeUDNS query message size (see below). response-sizeUDNS response message size (see below). response-processing -dataMData on response processing. Map of type ResponseProcessingData, see . query-extendedMExtended Query data. Map of type QueryResponseExtended, see . response-extendedMExtended Response data. Map of type QueryResponseExtended, see . The query-size and response-size fields hold the DNS message size. For UDP this is the size of the UDP payload that contained the DNS message. For TCP it is the size of the DNS message as specified in the two-byte message length header. Trailing bytes in UDP queries are routinely observed in traffic to authoritative servers and this value allows a calculation of how many trailing bytes were present.

Information on the server processing that produced the response. A map containing the following: Field M T Description bailiwick-indexUThe index in the name-rdata array of the owner name for the response bailiwick. See . processing-flagsUFlags relating to response processing. Bit 0. 1 if the response came from cache.

Extended data on the Q/R data item. Each item in the map is present only if collection of the relevant details is configured. A map containing the following items: Field M T Description question-indexUThe index in the qlist array of the entry listing any second and subsequent Questions in the Question section for the Query or Response. See . answer-indexUThe index in the rrlist array of the entry listing the Answer Resource Record sections for the Query or Response. See . authority-indexUThe index in the rrlist array of the entry listing the Authority Resource Record sections for the Query or Response. See . additional-indexUThe index in the rrlist array of the entry listing the Additional Resource Record sections for the Query or Response. See . Note that Query OPT RR data can be optionally stored in the QuerySignature.

Counts of various IP related events relating to traffic with individual client addresses. A map containing the following: Field M T Description ae-typeXUThe type of event. The following events types are currently defined: 0. TCP reset. 1. ICMP time exceeded. 2. ICMP destination unreachable. 3. ICMPv6 time exceeded. 4. ICMPv6 destination unreachable. 5. ICMPv6 packet too big. ae-codeUA code relating to the event. For ICMP or ICMPv6 events, this MUST be the ICMP or ICMPv6 code. For other events the contents are undefined. ae-address-indexXUThe index in the ip-address array of the client address. See . ae-countXUThe number of occurrences of this event during the block collection period.

Details of malformed messages. A map containing the following: Field M T Description time-offsetUMessage timestamp as an offset in ticks (see ) from earliest-time. client-address-indexUThe index in the ip-address array of the client IP address. See . client-portUThe client port. message-data-indexUThe index in the malformed-message-data array of the message data for this message. See .

The C-DNS file preamble includes a file format version; a major and minor version number are required fields. The document defines version 1.0 of the C-DNS specification. This section describes the intended use of these version numbers in future specifications. It is noted that version 1.0 includes many optional fields and therefore consumers of version 1.0 should be inherently robust to parsing files with variable data content. Within a major version, a new minor version MUST be a strict superset of the previous minor version, with no semantic changes to existing fields. New keys MAY be added to existing maps, and new maps MAY be added. A consumer capable of reading a particular major.minor version MUST also be capable of reading all previous minor versions of the same major version. It SHOULD also be capable of parsing all subsequent minor versions ignoring any keys or maps that it does not recognise. A new major version indicates changes to the format that are not backwards compatible with previous major versions. A consumer capable of only reading a particular major version (greater than 1) is not required to and has no expectation to be capable of reading a previous major version.

It is possible to re-construct PCAP files from the C-DNS format in a lossy fashion. Some of the issues with reconstructing both the DNS payload and the full packet stream are outlined here. The reconstruction depends on whether or not all the optional sections of both the query and response were captured in the C-DNS file. Clearly, if they were not all captured, the reconstruction will be imperfect. Even if all sections of the response were captured, one cannot reconstruct the DNS response payload exactly due to the fact that some DNS names in the message on the wire may have been compressed. discusses this is more detail. Some transport information is not captured in the C-DNS format. For example, the following aspects of the original packet stream cannot be re-constructed from the C-DNS format: IP fragmentation TCP stream information: Multiple DNS messages may have been sent in a single TCP segment A DNS payload may have been split across multiple TCP segments Multiple DNS messages may have been sent on a single TCP session Malformed DNS messages if the wire format is not recorded Any Non-DNS messages that were in the original packet stream e.g. ICMP Simple assumptions can be made on the reconstruction: fragmented and DNS-over-TCP messages can be reconstructed into single packets and a single TCP session can be constructed for each TCP packet. Additionally, if malformed messages and Non-DNS packets are captured separately, they can be merged with packet captures reconstructed from C-DNS to produce a more complete packet stream.

All the names stored in the C-DNS format are full domain names; no name compression is used on the individual names within the format. Therefore when reconstructing a packet, name compression must be used in order to reproduce the on the wire representation of the packet. name compression works by substituting trailing sections of a name with a reference back to the occurrence of those sections earlier in the message. Not all name server software uses the same algorithm when compressing domain names within the responses. Some attempt maximum recompression at the expense of runtime resources, others use heuristics to balance compression and speed and others use different rules for what is a valid compression target. This means that responses to the same question from different name server software which match in terms of DNS payload content (header, counts, RRs with name compression removed) do not necessarily match byte-for-byte on the wire. Therefore, it is not possible to ensure that the DNS response payload is reconstructed byte-for-byte from C-DNS data. However, it can at least, in principle, be reconstructed to have the correct payload length (since the original response length is captured) if there is enough knowledge of the commonly implemented name compression algorithms. For example, a simplistic approach would be to try each algorithm in turn to see if it reproduces the original length, stopping at the first match. This would not guarantee the correct algorithm has been used as it is possible to match the length whilst still not matching the on the wire bytes but, without further information added to the C-DNS data, this is the best that can be achieved. presents an example of two different compression algorithms used by well-known name server software.

This section describes a non-normative proposed algorithm for the processing of a captured stream of DNS queries and responses and production of a stream of query/response items, matching queries/responses where possible. For the purposes of this discussion, it is assumed that the input has been pre-processed such that: All IP fragmentation reassembly, TCP stream reassembly, and so on, has already been performed. Each message is associated with transport metadata required to generate the Primary ID (see ). Each message has a well-formed DNS header of 12 bytes and (if present) the first Question in the Question section can be parsed to generate the Secondary ID (see below). As noted earlier, this requirement can result in a malformed query being removed in the pre-processing stage, but the correctly formed response with RCODE of FORMERR being present. DNS messages are processed in the order they are delivered to the implementation. It should be noted that packet capture libraries do not necessarily provide packets in strict chronological order. This can, for example, arise on multi-core platforms where packets arriving at a network device are processed by different cores. On systems where this behaviour has been observed, the timestamps associated with each packet are consistent; queries always have a timestamp prior to the response timestamp. However, the order in which these packets appear in the packet capture stream is not necessarily strictly chronological; a response can appear in the capture stream before the query that provoked the response. For this discussion, this non-chronological delivery is termed "skew". In the presence of skew, a response packets can arrive for matching before the corresponding query. To avoid generating false instances of responses without a matching query, and queries without a matching response, the matching algorithm must take account of the possibility of skew.

A schematic representation of the algorithm for matching Q/R data items is shown in Figure 3. It takes individual DNS query or response messages as input, and outputs matched Q/R items. The numbers in the figure identify matching operations listed in Table 1. Specific details of the algorithm, for example queues, timers and identifiers, are given in the following sections.

.----------------------. | Process next message |<------------------+ `----------------------' | | | +------------------------------+ | | Generate message identifiers | | +------------------------------+ | | | Response | Query | +--------------< >---------------+ | | | | +--------------------+ +--------------------+ | | Find earliest QR | | Create QR item [2] | | | item in OFIFO [1] | +--------------------+ | +--------------------+ | | | +---------------+ | Match | No match | Append new QR | | +--------< >------+ | item to OFIFO | | | | +---------------+ | +-----------+ +--------+ | | | Update QR | | Add to | +-------------------+ | | item [3] | | RFIFO | | Find earliest QR | | +-----------+ +--------+ | item in RFIFO [1] | | | | +-------------------+ | +-----------------+ | | | | | | +----------------+ Match | No match | | | Remove R |-------< >-----+ | | | from RFIFO [3] | | | | +----------------+ | | | | | | +--------------+-----------------------+ | | | +----------------------------------------------+ | | Update all timed out (QT) OFIFO QR items [4] | | +----------------------------------------------+ | | | +--------------------------------+ | | Remove all timed out (ST) R | | | from RFIFO, create QR item [5] | | +--------------------------------+ | ____________________|_______________________ | / / | / Remove all consecutive done entries from /-------+ / front of OFIFO for further processing / /____________________________________________/

Ref Operation [1]Find earliest QR item in FIFO where: * QR.done = false * QR.Q.PrimaryID == R.PrimaryID and, if both QR.Q and R have SecondaryID: * QR.Q.SecondaryID == R.SecondaryID [2]Set: QR.Q := Q QR.R := nil QR.done := false [3]Set: QR.R := R QR.done := true [4]Set: QR.done := true [5]Set: QR.Q := nil QR.R := R QR.done := true

A Primary ID is constructed for each message. It is composed of the following data: Source IP Address Destination IP Address Source Port Destination Port Transport DNS Message ID

If present, the first Question in the Question section is used as a secondary ID for each message. Note that there may be well formed DNS queries that have a QDCOUNT of 0, and some responses may have a QDCOUNT of 0 (for example, responses with RCODE=FORMERR or NOTIMP). In this case the secondary ID is not used in matching.

Query timeout, QT. A query arrives with timestamp t1. If no response matching that query has arrived before other input arrives timestamped later than (t1 + QT), a query/response item containing only a query item is recorded. The query timeout value is typically of the order of 5 seconds. Skew timeout, ST. A response arrives with timestamp t2. If a response has not been matched by a query before input arrives timestamped later than (t2 + ST), a query/response item containing only a response is recorded. The skew timeout value is typically a few microseconds.

The algorithm is designed to handle the following input data: Multiple queries with the same Primary ID (but different Secondary ID) arriving before any responses for these queries are seen. Multiple queries with the same Primary and Secondary ID arriving before any responses for these queries are seen. Queries for which no later response can be found within the specified timeout. Responses for which no previous query can be found within the specified timeout.

For cases 1 and 2 listed in the above requirements, it is not possible to unambiguously match queries with responses. This algorithm chooses to match to the earliest query with the correct Primary and Secondary ID.

The algorithm employs two FIFO queues: OFIFO, an output FIFO containing Q/R items in chronological order, RFIFO, a FIFO holding responses without a matching query in order of arrival.

The output is a list of Q/R data items. Both the Query and Response elements are optional in these items, therefore Q/R data items have one of three types of content: A matched pair of query and response messages A query message with no response A response message with no query The timestamp of a list item is that of the query for cases 1 and 2 and that of the response for case 3.

When ending capture, all items in the responses FIFO are timed out immediately, generating response-only entries to the Q/R data item FIFO. These and all other remaining entries in the Q/R data item FIFO should be treated as timed out queries.

Whilst this document makes no specific recommendations with respect to Canonical CBOR (see Section 3.9 of ) the following guidance may be of use to implementors. Adherence to the first two rules given in Section 3.9 of will minimise file sizes. Adherence to the last two rules given in Section 3.9 of for all maps and arrays would unacceptably constrain implementations, for example, in the use case of real-time data collection in constrained environments where outputting block tables after query/response data and allowing indefinite length maps and arrays could reduce memory requirements.

When decoding C-DNS data some of the items required for a particular function that the consumer wishes to perform may be missing. Consumers should consider providing configurable default values to be used in place of the missing values in their output.

A DNS query message in a UDP or TCP payload can be followed by some additional (spurious) bytes, which are not stored in C-DNS. When DNS traffic is sent over TCP, each message is prefixed with a two byte length field which gives the message length, excluding the two byte length field. In this context, trailing bytes can occur in two circumstances with different results: The number of bytes consumed by fully parsing the message is less than the number of bytes given in the length field (i.e. the length field is incorrect and too large). In this case, the surplus bytes are considered trailing bytes in an analogous manner to UDP and recorded as such. If only this case occurs it is possible to process a packet containing multiple DNS messages where one or more has trailing bytes. There are surplus bytes between the end of a well-formed message and the start of the length field for the next message. In this case the first of the surplus bytes will be processed as the first byte of the next length field, and parsing will proceed from there, almost certainly leading to the next and any subsequent messages in the packet being considered malformed. This will not generate a trailing bytes record for the processed well-formed message.

Implementations should consider providing a configurable maximum RDATA size for capture, for example, to avoid memory issues when confronted with large XFR records.

The preamble to each block includes a timestamp of the earliest record in the block. As described in , the timestamp is an array of 2 unsigned integers. The first is a POSIX time_t . Consumers of C-DNS should be aware of this as it excludes leap seconds and therefore may cause minor anomalies in the data e.g. when calculating query throughput.

[Note to RFC Editor: please remove this section and reference to prior to publication.] This section records the status of known implementations of the protocol defined by this specification at the time of posting of this Internet-Draft, and is based on a proposal described in . The description of implementations in this section is intended to assist the IETF in its decision processes in progressing drafts to RFCs. Please note that the listing of any individual implementation here does not imply endorsement by the IETF. Furthermore, no effort has been spent to verify the information presented here that was supplied by IETF contributors. This is not intended as, and must not be construed to be, a catalog of available implementations or their features. Readers are advised to note that other implementations may exist. According to , "this will allow reviewers and working groups to assign due consideration to documents that have the benefit of running code, which may serve as evidence of valuable experimentation and feedback that have made the implemented protocols more mature. It is up to the individual working groups to use this information as they see fit".

ICANN/Sinodun IT have developed an open source implementation called DNS-STATS Compactor. The Compactor is a suite of tools which can capture DNS traffic (from either a network interface or a PCAP file) and store it in the Compacted-DNS (C-DNS) file format. PCAP files for the captured traffic can also be reconstructed. See Compactor. This implementation: covers the whole of the specification described in the -03 draft with the exception of support for malformed messages and pico second time resolution. (Note: this implementation does allow malformed messages to be recorded separately in a PCAP file). is released under the Mozilla Public License Version 2.0. has a users mailing list available, see dns-stats-users. There is also some discussion of issues encountered during development available at Compressing Pcap Files and Packet Capture. This information was last updated on 3rd of May 2018.

IANA is requested to create a registry "C-DNS DNS Capture Format" containing the subregistries defined in sections to inclusive. In all cases, new entries may be added to the subregistries by Expert Review as defined in . Experts are expected to exercise their own expert judgement, and should consider the following general guidelines in addition to any guidelines given particular to a subregistry. There should be a real and compelling use for any new value. Values assigned should be carefully chosen to minimise storage requirements for common cases.

IANA is requested to create a registry "C-DNS Transports" of C-DNS transport type identifiers. The primary purpose of this registry is to provide unique identifiers for all transports used for DNS queries. The following note is included in this registry: "In version 1.0 of C-DNS [[this RFC]], there is a field to identify the type of DNS transport. This field is 4 bits in size." The initial contents of the registry are as follows - see sections and of [[this RFC]]: Identifier Name Reference 0UDP[[this RFC]] 1TCP[[this RFC]] 2TLS[[this RFC]] 3DTLS[[this RFC]] 4DoH[[this RFC]] 5-15Unassigned Expert reviewers should take the following points into consideration: Is the requested DNS transport described by a Standards Track RFC?

IANA is requested to create a registry "C-DNS Storage Flags" of C-DNS data storage flags. The primary purpose of this registry is to provide indicators giving hints on processing of the data stored. The following note is included in this registry: "In version 1.0 of C-DNS [[this RFC]], there is a field describing attributes of the data recorded. The field is a CBOR unsigned integer holding bit flags." The initial contents of the registry are as follows - see section of [[this RFC]]: Bit Name Description Reference 0anonymised-dataThe data has been anonymised.[[this RFC]] 1sampled-dataThe data is sampled data.[[this RFC]] 2normalized-namesNames in the data have been normalized.[[this RFC]] 3-63Unassigned

IANA is requested to create a registry "C-DNS Response Flags" of C-DNS response processing flags. The primary purpose of this registry is to provide indicators giving hints on the generation of a particular response. The following note is included in this registry: "In version 1.0 of C-DNS [[this RFC]], there is a field describing attributes of the responses recorded. The field is a CBOR unsigned integer holding bit flags." The initial contents of the registry are as follows - see section of [[this RFC]]: Bit Name Description Reference 0from-cacheThe response came from cache.[[this RFC]] 1-63Unassigned

IANA is requested to create a registry "C-DNS Address Event Types" of C-DNS AddressEvent types. The primary purpose of this registry is to provide unique identifiers of different types of C-DNS address events, and so specify the contents of the optional companion field ae-code for each type. The following note is included in this registry: "In version 1.0 of C-DNS [[this RFC]], there is a field identify types of the events related to client addresses. This field is a CBOR unsigned integer. There is a related optional field ae-code, which, if present, holds an additional CBOR unsigned integer giving additional information specific to the event type." The initial contents of the registry are as follows - see section : Identifier Event Type ae-code contents Reference 0TCP resetNone[[this RFC]] 1ICMP time exceededICMP code [[this RFC]] 2ICMP destination unreachableICMP code [[this RFC]] 3ICMPv6 time exceededICMPv6 code [[this RFC]] 4ICMPv6 destination unreachableICMPv6 code [[this RFC]] 5ICMPv6 packet too bigICMPv6 code [[this RFC]] >5Unassigned Expert reviewers should take the following points into consideration: ae-code contents must be defined for a type, or if not appropriate specified as None. A specification of None requires less storage, and is therefore preferred.

Any control interface MUST perform authentication and encryption. Any data upload MUST be authenticated and encrypted.

Storage of DNS traffic by operators in PCAP and other formats is a long standing and widespread practice. Section 2.5 of is an analysis of the risks to Internet users of the storage of DNS traffic data in servers (recursive resolvers, authoritative and rogue servers). Section 5.2 of describes mitigations for those risks for data stored on recursive resolvers (but which could by extension apply to authoritative servers). These include data handling practices and methods for data minimization, IP address pseudonymization and anonymization. Appendix B of that document presents an analysis of 7 published anonymization processes. In addition, RSSAC have recently published RSSAC04: " Recommendations on Anonymization Processes for Source IP Addresses Submitted for Future Analysis”. The above analyses consider full data capture (e.g using PCAP) as a baseline for privacy considerations and therefore this format specification introduces no new user privacy issues beyond those of full data capture (which are quite severe). It does provides mechanisms to selectively record only certain fields at the time of data capture to improve user privacy and to explicitly indicate that data is sampled and or anonymized. It also provide flags to indicate if data normalization has been performed; data normalization increases user privacy by reducing the potential for fingerprinting individuals, however, a trade-off is potentially reducing the capacity to identify attack traffic via query name signatures. Operators should carefully consider their operational requirements and privacy policies and SHOULD capture at source the minimum user data required to meet their needs.

The authors wish to thank CZ.NIC, in particular Tomas Gavenciak, for many useful discussions on binary formats, compression and packet matching. Also Jan Vcelak and Wouter Wijngaards for discussions on name compression and Paul Hoffman for a detailed review of the document and the C-DNS CDDL. Thanks also to Robert Edmonds, Jerry Lundström, Richard Gibson, Stephane Bortzmeyer and many other members of DNSOP for review. Also, Miek Gieben for mmark

draft-ietf-dnsop-dns-capture-format-10 Add IANA Considerations Convert graph in C.6 to table draft-ietf-dnsop-dns-capture-format-09 Editorial changes arising from IESG review *-transport-flags and may be mandatory in some configurations Mark fields that are conditionally mandatory Change `promisc' flag CDDL data type to boolean Add ranges to configuration quantities where appropriate draft-ietf-dnsop-dns-capture-format-08 Convert diagrams to ASCII Describe versioning Fix unused group warning in CDDL draft-ietf-dnsop-dns-capture-format-07 Resolve outstanding questions and TODOs Make RR RDATA optional Update matching diagram and explain skew Add count of discarded messages to block statistics Editorial clarifications and improvements draft-ietf-dnsop-dns-capture-format-06 Correct BlockParameters type to map Make RR ttl optional Add storage flag indicating name normalization Add storage parameter fields for sampling and anonymization methods Editorial clarifications and improvements draft-ietf-dnsop-dns-capture-format-05 Make all data items in Q/R, QuerySignature and Malformed Message arrays optional Re-structure the FilePreamble and ConfigurationParameters into BlockParameters BlockParameters has separate Storage and Collection Parameters Storage Parameters includes information on what optional fields are present, and flags specifying anonymization or sampling Addresses can now be stored as prefixes. Switch to using a variable sub-second timing granularity Add response bailiwick and query response type Add specifics of how to record malformed messages Add implementation guidance Improve terminology and naming consistency draft-ietf-dnsop-dns-capture-format-04 Correct query-d0 to query-do in CDDL Clarify that map keys are unsigned integers Add Type to Class/Type table Clarify storage format in section 7.12 draft-ietf-dnsop-dns-capture-format-03 Added an Implementation Status section draft-ietf-dnsop-dns-capture-format-02 Update qr_data_format.png to match CDDL Editorial clarifications and improvements draft-ietf-dnsop-dns-capture-format-01 Many editorial improvements by Paul Hoffman Included discussion of malformed message handling Improved Appendix C on Comparison of Binary Formats Now using C-DNS field names in the tables in section 8 A handful of new fields included (CDDL updated) Timestamps now include optional picoseconds Added details of block statistics draft-ietf-dnsop-dns-capture-format-00 Changed dnstap.io to dnstap.info qr_data_format.png was cut off at the bottom Update authors address Improve wording in Abstract Changed DNS-STAT to C-DNS in CDDL Set the format version in the CDDL Added a TODO: Add block statistics Added a TODO: Add extend to support pico/nano. Also do this for Time offset and Response delay Added a TODO: Need to develop optional representation of malformed messages within C-DNS and what this means for packet matching. This may influence which fields are optional in the rest of the representation. Added section on design goals to Introduction Added a TODO: Can Class be optimised? Should a class of IN be inferred if not present? draft-dickinson-dnsop-dns-capture-format-00 Initial commit

tcpdump.org tcpdump.org The Open Group IEEE DNS-OARC DNS-OARC dnstap.info Verisign DNS-OARC IANA IANA IANA .SE - The Internet Infrastructure Foundation tcpdump.org Muenster Univ. of Appl. Sciences

tuexen@fh-muenster.de

Politecnico di Torino

fulvio.risso@polito.it

Airbus DS CyberSecurity

jasper@packet-foo.com

Wireshark

gerald@wireshark.org

guy@alum.mit.edu

IANA IANA IANA

This appendix gives a CDDL specification for C-DNS. CDDL does not permit a range of allowed values to be specified for a bitfield. Where necessary, those values are given as a CDDL group, but the group definition is commented out to prevent CDDL tooling from warning that the group is unused.

; CDDL specification of the file format for C-DNS, ; which describes a collection of DNS messages and ; traffic meta-data. ; ; The overall structure of a file. ; File = [ file-type-id : "C-DNS", file-preamble : FilePreamble, file-blocks : [* Block], ] ; ; The file preamble. ; FilePreamble = { major-format-version => 1, minor-format-version => 0, ? private-version => uint, block-parameters => [+ BlockParameters], } major-format-version = 0 minor-format-version = 1 private-version = 2 block-parameters = 3 BlockParameters = { storage-parameters => StorageParameters, ? collection-parameters => CollectionParameters, } storage-parameters = 0 collection-parameters = 1 IPv6PrefixLength = 1..128 IPv4PrefixLength = 1..32 OpcodeRange = 0..15 RRTypeRange = 0..65535 StorageParameters = { ticks-per-second => uint, max-block-items => uint, storage-hints => StorageHints, opcodes => [+ OpcodeRange], rr-types => [+ RRTypeRange], ? storage-flags => StorageFlags, ? client-address-prefix-ipv4 => IPv4PrefixLength, ? client-address-prefix-ipv6 => IPv6PrefixLength, ? server-address-prefix-ipv4 => IPv4PrefixLength, ? server-address-prefix-ipv6 => IPv6PrefixLength, ? sampling-method => tstr, ? anonymisation-method => tstr, } ticks-per-second = 0 max-block-items = 1 storage-hints = 2 opcodes = 3 rr-types = 4 storage-flags = 5 client-address-prefix-ipv4 = 6 client-address-prefix-ipv6 = 7 server-address-prefix-ipv4 = 8 server-address-prefix-ipv6 = 9 sampling-method = 10 anonymisation-method = 11 ; A hint indicates if the collection method will output the ; item or will ignore the item if present. StorageHints = { query-response-hints => QueryResponseHints, query-response-signature-hints => QueryResponseSignatureHints, rr-hints => RRHints, other-data-hints => OtherDataHints, } query-response-hints = 0 query-response-signature-hints = 1 rr-hints = 2 other-data-hints = 3 QueryResponseHintValues = &( time-offset : 0, client-address-index : 1, client-port : 2, transaction-id : 3, qr-signature-index : 4, client-hoplimit : 5, response-delay : 6, query-name-index : 7, query-size : 8, response-size : 9, response-processing-data : 10, query-question-sections : 11, ; Second & subsequent ; questions query-answer-sections : 12, query-authority-sections : 13, query-additional-sections : 14, response-answer-sections : 15, response-authority-sections : 16, response-additional-sections : 17, ) QueryResponseHints = uint .bits QueryResponseHintValues QueryResponseSignatureHintValues = &( server-address : 0, server-port : 1, qr-transport-flags : 2, qr-type : 3, qr-sig-flags : 4, query-opcode : 5, dns-flags : 6, query-rcode : 7, query-class-type : 8, query-qdcount : 9, query-ancount : 10, query-arcount : 11, query-nscount : 12, query-edns-version : 13, query-udp-size : 14, query-opt-rdata : 15, response-rcode : 16, ) QueryResponseSignatureHints = uint .bits QueryResponseSignatureHintValues RRHintValues = &( ttl : 0, rdata-index : 1, ) RRHints = uint .bits RRHintValues OtherDataHintValues = &( malformed-messages : 0, address-event-counts : 1, ) OtherDataHints = uint .bits OtherDataHintValues StorageFlagValues = &( anonymised-data : 0, sampled-data : 1, normalized-names : 2, ) StorageFlags = uint .bits StorageFlagValues ; Hints for later analysis. VLANIdRange = 1..4094 CollectionParameters = { ? query-timeout => uint, ? skew-timeout => uint, ? snaplen => uint, ? promisc => bool, ? interfaces => [+ tstr], ? server-addresses => [+ IPAddress], ? vlan-ids => [+ VLANIdRange], ? filter => tstr, ? generator-id => tstr, ? host-id => tstr, } query-timeout = 0 skew-timeout = 1 snaplen = 2 promisc = 3 interfaces = 4 server-addresses = 5 vlan-ids = 6 filter = 7 generator-id = 8 host-id = 9 ; ; Data in the file is stored in Blocks. ; Block = { block-preamble => BlockPreamble, ? block-statistics => BlockStatistics, ; Much of this ; could be derived ? block-tables => BlockTables, ? query-responses => [+ QueryResponse], ? address-event-counts => [+ AddressEventCount], ? malformed-messages => [+ MalformedMessage], } block-preamble = 0 block-statistics = 1 block-tables = 2 query-responses = 3 address-event-counts = 4 malformed-messages = 5 ; ; The (mandatory) preamble to a block. ; BlockPreamble = { ? earliest-time => Timestamp, ? block-parameters-index => uint .default 0, } earliest-time = 0 block-parameters-index = 1 ; Ticks are subsecond intervals. The number of ticks in a second is ; file/block metadata. Signed and unsigned tick types are defined. ticks = int uticks = uint Timestamp = [ timestamp-secs : uint, timestamp-uticks : uticks, ] ; ; Statistics about the block contents. ; BlockStatistics = { ? processed-messages => uint, ? qr-data-items => uint, ? unmatched-queries => uint, ? unmatched-responses => uint, ? discarded-opcode => uint, ? malformed-items => uint, } processed-messages = 0 qr-data-items = 1 unmatched-queries = 2 unmatched-responses = 3 discarded-opcode = 4 malformed-items = 5 ; ; Tables of common data referenced from records in a block. ; BlockTables = { ? ip-address => [+ IPAddress], ? classtype => [+ ClassType], ? name-rdata => [+ bstr], ; Holds both Names ; and RDATA ? qr-sig => [+ QueryResponseSignature], ? QuestionTables, ? RRTables, ? malformed-message-data => [+ MalformedMessageData], } ip-address = 0 classtype = 1 name-rdata = 2 qr-sig = 3 qlist = 4 qrr = 5 rrlist = 6 rr = 7 malformed-message-data = 8 IPv4Address = bstr .size 4 IPv6Address = bstr .size 16 IPAddress = IPv4Address / IPv6Address ClassType = { type => uint, class => uint, } type = 0 class = 1 QueryResponseSignature = { ? server-address-index => uint, ? server-port => uint, ? qr-transport-flags => QueryResponseTransportFlags, ? qr-type => QueryResponseType, ? qr-sig-flags => QueryResponseFlags, ? query-opcode => uint, ? qr-dns-flags => DNSFlags, ? query-rcode => uint, ? query-classtype-index => uint, ? query-qd-count => uint, ? query-an-count => uint, ? query-ns-count => uint, ? query-ar-count => uint, ? edns-version => uint, ? udp-buf-size => uint, ? opt-rdata-index => uint, ? response-rcode => uint, } server-address-index = 0 server-port = 1 qr-transport-flags = 2 qr-type = 3 qr-sig-flags = 4 query-opcode = 5 qr-dns-flags = 6 query-rcode = 7 query-classtype-index = 8 query-qd-count = 9 query-an-count = 10 query-ns-count = 12 query-ar-count = 12 edns-version = 13 udp-buf-size = 14 opt-rdata-index = 15 response-rcode = 16 ; Transport gives the values that may appear in bits 1..4 of ; TransportFlags. There is currently no way to express this in ; CDDL, so Transport is unused. To avoid confusion when used ; with CDDL tools, it is commented out. ; ; Transport = &( ; udp : 0, ; tcp : 1, ; tls : 2, ; dtls : 3, ; doh : 4, ; ) TransportFlagValues = &( ip-version : 0, ; 0=IPv4, 1=IPv6 ) / (1..4) TransportFlags = uint .bits TransportFlagValues QueryResponseTransportFlagValues = &( query-trailingdata : 5, ) / TransportFlagValues QueryResponseTransportFlags = uint .bits QueryResponseTransportFlagValues QueryResponseType = &( stub : 0, client : 1, resolver : 2, auth : 3, forwarder : 4, tool : 5, ) QueryResponseFlagValues = &( has-query : 0, has-reponse : 1, query-has-opt : 2, response-has-opt : 3, query-has-no-question : 4, response-has-no-question: 5, ) QueryResponseFlags = uint .bits QueryResponseFlagValues DNSFlagValues = &( query-cd : 0, query-ad : 1, query-z : 2, query-ra : 3, query-rd : 4, query-tc : 5, query-aa : 6, query-do : 7, response-cd: 8, response-ad: 9, response-z : 10, response-ra: 11, response-rd: 12, response-tc: 13, response-aa: 14, ) DNSFlags = uint .bits DNSFlagValues QuestionTables = ( qlist => [+ QuestionList], qrr => [+ Question] ) QuestionList = [+ uint] ; Index of Question Question = { ; Second and subsequent questions name-index => uint, ; Index to a name in the ; name-rdata table classtype-index => uint, } name-index = 0 classtype-index = 1 RRTables = ( rrlist => [+ RRList], rr => [+ RR] ) RRList = [+ uint] ; Index of RR RR = { name-index => uint, ; Index to a name in the ; name-rdata table classtype-index => uint, ? ttl => uint, ? rdata-index => uint, ; Index to RDATA in the ; name-rdata table } ; Other map key values already defined above. ttl = 2 rdata-index = 3 MalformedMessageData = { ? server-address-index => uint, ? server-port => uint, ? mm-transport-flags => TransportFlags, ? mm-payload => bstr, } ; Other map key values already defined above. mm-transport-flags = 2 mm-payload = 3 ; ; A single query/response pair. ; QueryResponse = { ? time-offset => uticks, ; Time offset from ; start of block ? client-address-index => uint, ? client-port => uint, ? transaction-id => uint, ? qr-signature-index => uint, ? client-hoplimit => uint, ? response-delay => ticks, ? query-name-index => uint, ? query-size => uint, ; DNS size of query ? response-size => uint, ; DNS size of response ? response-processing-data => ResponseProcessingData, ? query-extended => QueryResponseExtended, ? response-extended => QueryResponseExtended, } time-offset = 0 client-address-index = 1 client-port = 2 transaction-id = 3 qr-signature-index = 4 client-hoplimit = 5 response-delay = 6 query-name-index = 7 query-size = 8 response-size = 9 response-processing-data = 10 query-extended = 11 response-extended = 12 ResponseProcessingData = { ? bailiwick-index => uint, ? processing-flags => ResponseProcessingFlags, } bailiwick-index = 0 processing-flags = 1 ResponseProcessingFlagValues = &( from-cache : 0, ) ResponseProcessingFlags = uint .bits ResponseProcessingFlagValues QueryResponseExtended = { ? question-index => uint, ; Index of QuestionList ? answer-index => uint, ; Index of RRList ? authority-index => uint, ? additional-index => uint, } question-index = 0 answer-index = 1 authority-index = 2 additional-index = 3 ; ; Address event data. ; AddressEventCount = { ae-type => &AddressEventType, ? ae-code => uint, ae-address-index => uint, ae-count => uint, } ae-type = 0 ae-code = 1 ae-address-index = 2 ae-count = 3 AddressEventType = ( tcp-reset : 0, icmp-time-exceeded : 1, icmp-dest-unreachable : 2, icmpv6-time-exceeded : 3, icmpv6-dest-unreachable: 4, icmpv6-packet-too-big : 5, ) ; ; Malformed messages. ; MalformedMessage = { ? time-offset => uticks, ; Time offset from ; start of block ? client-address-index => uint, ? client-port => uint, ? message-data-index => uint, } ; Other map key values already defined above. message-data-index = 3

The basic algorithm, which follows the guidance in , is simply to collect each name, and the offset in the packet at which it starts, during packet construction. As each name is added, it is offered to each of the collected names in order of collection, starting from the first name. If labels at the end of the name can be replaced with a reference back to part (or all) of the earlier name, and if the uncompressed part of the name is shorter than any compression already found, the earlier name is noted as the compression target for the name. The following tables illustrate the process. In an example packet, the first name is foo.example. N Name Uncompressed Compression Target 1foo.example The next name added is bar.example. This is matched against foo.example. The example part of this can be used as a compression target, with the remaining uncompressed part of the name being bar. N Name Uncompressed Compression Target 1foo.example 2bar.examplebar1 + offset to example The third name added is www.bar.example. This is first matched against foo.example, and as before this is recorded as a compression target, with the remaining uncompressed part of the name being www.bar. It is then matched against the second name, which again can be a compression target. Because the remaining uncompressed part of the name is www, this is an improved compression, and so it is adopted. N Name Uncompressed Compression Target 1foo.example 2bar.examplebar1 + offset to example 3www.bar.examplewww2 As an optimization, if a name is already perfectly compressed (in other words, the uncompressed part of the name is empty), then no further names will be considered for compression.

Using the above basic algorithm the packet lengths of responses generated by NSD can be matched almost exactly. At the time of writing, a tiny number (<.01%) of the reconstructed packets had incorrect lengths.

The Knot Authoritative name server uses different compression behavior, which is the result of internal optimization designed to balance runtime speed with compression size gains. In brief, and omitting complications, Knot Authoritative will only consider the QNAME and names in the immediately preceding RR section in an RRSET as compression targets. A set of smart heuristics as described below can be implemented to mimic this and while not perfect it produces output nearly, but not quite, as good a match as with NSD. The heuristics are: A match is only perfect if the name is completely compressed AND the TYPE of the section in which the name occurs matches the TYPE of the name used as the compression target. If the name occurs in RDATA: If the compression target name is in a query, then only the first RR in an RRSET can use that name as a compression target. The compression target name MUST be in RDATA. The name section TYPE must match the compression target name section TYPE. The compression target name MUST be in the immediately preceding RR in the RRSET. Using this algorithm less than 0.1% of the reconstructed packets had incorrect lengths.

In sample traffic collected on a root name server around 2-4% of responses generated by Knot had different packet lengths to those produced by NSD.

Several binary serialisation formats were considered, and for completeness were also compared to JSON. Apache Avro. Data is stored according to a pre-defined schema. The schema itself is always included in the data file. Data can therefore be stored untagged, for a smaller serialisation size, and be written and read by an Avro library. At the time of writing, Avro libraries are available for C, C++, C#, Java, Python, Ruby and PHP. Optionally tools are available for C++, Java and C# to generate code for encoding and decoding. Google Protocol Buffers. Data is stored according to a pre-defined schema. The schema is used by a generator to generate code for encoding and decoding the data. Data can therefore be stored untagged, for a smaller serialisation size. The schema is not stored with the data, so unlike Avro cannot be read with a generic library. Code must be generated for a particular data schema to read and write data using that schema. At the time of writing, the Google code generator can currently generate code for encoding and decoding a schema for C++, Go, Java, Python, Ruby, C#, Objective-C, Javascript and PHP. CBOR. Defined in , this serialisation format is comparable to JSON but with a binary representation. It does not use a pre-defined schema, so data is always stored tagged. However, CBOR data schemas can be described using CDDL and tools exist to verify data files conform to the schema. CBOR is a simple format, and simple to implement. At the time of writing, the CBOR website lists implementations for 16 languages. Avro and Protocol Buffers both allow storage of untagged data, but because they rely on the data schema for this, their implementation is considerably more complex than CBOR. Using Avro or Protocol Buffers in an unsupported environment would require notably greater development effort compared to CBOR. A test program was written which reads input from a PCAP file and writes output using one of two basic structures; either a simple structure, where each query/response pair is represented in a single record entry, or the C-DNS block structure. The resulting output files were then compressed using a variety of common general-purpose lossless compression tools to explore the compressibility of the formats. The compression tools employed were: snzip. A command line compression tool based on the Google Snappy library. lz4. The command line compression tool from the reference C LZ4 implementation. gzip. The ubiquitous GNU zip tool. zstd. Compression using the Zstandard algorithm. xz. A popular compression tool noted for high compression. In all cases the compression tools were run using their default settings. Note that this draft does not mandate the use of compression, nor any particular compression scheme, but it anticipates that in practice output data will be subject to general-purpose compression, and so this should be taken into consideration. test.pcap, a 662Mb capture of sample data from a root instance was used for the comparison. The following table shows the formatted size and size after compression (abbreviated to Comp. in the table headers), together with the task resident set size (RSS) and the user time taken by the compression. File sizes are in Mb, RSS in kb and user time in seconds. Format File size Comp. Comp. size RSS User time PCAP661.87snzip212.4826961.26 lz4181.5863361.35 gzip153.46142818.20 zstd87.0735444.27 xz49.0997416160.79 JSON simple4113.92snzip603.7826565.72 lz4386.4256365.25 gzip271.11149273.00 zstd133.4332848.68 xz51.9897412600.74 Avro simple640.45snzip148.9826560.90 lz4111.9258280.99 gzip103.07154011.52 zstd49.0835242.50 xz22.879730890.34 CBOR simple764.82snzip164.5726641.11 lz4120.9858921.13 gzip110.61142812.88 zstd54.1432242.77 xz23.4397276111.48 PBuf simple749.51snzip167.1626601.08 lz4123.0958241.14 gzip112.05142412.75 zstd53.3933882.76 xz23.9997348106.47 JSON block519.77snzip106.1228120.93 lz4104.3460800.97 gzip57.97160412.70 zstd61.5133963.45 xz27.6797524169.10 Avro block60.45snzip48.3826880.20 lz448.7885400.22 gzip39.6215762.92 zstd29.6336121.25 xz18.289756425.81 CBOR block75.25snzip53.2726840.24 lz451.8880080.28 gzip41.1715484.36 zstd30.6134761.48 xz18.159755638.78 PBuf block67.98snzip51.1026360.24 lz452.3983040.24 gzip40.1915203.63 zstd31.6135761.40 xz17.949744033.99 The above results are discussed in the following sections.

An important first consideration is whether moving away from PCAP offers significant benefits. The simple binary formats are typically larger than PCAP, even though they omit some information such as Ethernet MAC addresses. But not only do they require less CPU to compress than PCAP, the resulting compressed files are smaller than compressed PCAP.

The intention of the block coding is to perform data de-duplication on query/response records within the block. The simple and block formats above store exactly the same information for each query/response record. This information is parsed from the DNS traffic in the input PCAP file, and in all cases each field has an identifier and the field data is typed. The data de-duplication on the block formats show an order of magnitude reduction in the size of the format file size against the simple formats. As would be expected, the compression tools are able to find and exploit a lot of this duplication, but as the de-duplication process uses knowledge of DNS traffic, it is able to retain a size advantage. This advantage reduces as stronger compression is applied, as again would be expected, but even with the strongest compression applied the block formatted data remains around 75% of the size of the simple format and its compression requires roughly a third of the CPU time.

Text data formats offer many advantages over binary formats, particularly in the areas of ad-hoc data inspection and extraction. It was therefore felt worthwhile to carry out a direct comparison, implementing JSON versions of the simple and block formats. Concentrating on JSON block format, the format files produced are a significant fraction of an order of magnitude larger than binary formats. The impact on file size after compression is as might be expected from that starting point; the stronger compression produces files that are 150% of the size of similarly compressed binary format, and require over 4x more CPU to compress.

Concentrating again on the block formats, all three produce format files that are close to an order of magnitude smaller that the original test.pcap file. CBOR produces the largest files and Avro the smallest, 20% smaller than CBOR. However, once compression is taken into account, the size difference narrows. At medium compression (with gzip), the size difference is 4%. Using strong compression (with xz) the difference reduces to 2%, with Avro the largest and Protocol Buffers the smallest, although CBOR and Protocol Buffers require slightly more compression CPU. The measurements presented above do not include data on the CPU required to generate the format files. Measurements indicate that writing Avro requires 10% more CPU than CBOR or Protocol Buffers. It appears, therefore, that Avro's advantage in compression CPU usage is probably offset by a larger CPU requirement in writing Avro.

The above assessments lead us to the choice of a binary format file using blocking. As noted previously, this draft anticipates that output data will be subject to compression. There is no compelling case for one particular binary serialisation format in terms of either final file size or machine resources consumed, so the choice must be largely based on other factors. CBOR was therefore chosen as the binary serialisation format for the reasons listed in .

Given the choice of a CBOR format using blocking, the question arises of what an appropriate default value for the maximum number of query/response pairs in a block should be. This has two components; what is the impact on performance of using different block sizes in the format file, and what is the impact on the size of the format file before and after compression. The following table addresses the performance question, showing the impact on the performance of a C++ program converting test.pcap to C-DNS. File size is in Mb, resident set size (RSS) in kb. Block size File size RSS User time 1000133.46612.2715.25 500089.85676.8214.99 1000076.87752.4014.53 2000067.86750.7514.49 4000061.88736.3014.29 8000058.08694.1614.28 16000055.94733.8414.44 32000054.41799.2013.97 Increasing block size, therefore, tends to increase maximum RSS a little, with no significant effect (if anything a small reduction) on CPU consumption. The following table demonstrates the effect of increasing block size on output file size for different compressions. Block size None snzip lz4 gzip zstd xz 1000133.4690.5290.0374.6544.7825.63 500089.8559.6959.4346.9937.3322.34 1000076.8750.3950.2838.9433.6221.09 2000067.8643.9143.9033.2432.6220.16 4000061.8839.6339.6929.4428.7219.52 8000058.0836.9337.0127.0526.2519.00 16000055.9435.1035.0625.4424.5619.63 32000054.4133.8733.7424.3623.4418.66 There is obviously scope for tuning the default block size to the compression being employed, traffic characteristics, frequency of output file rollover etc. Using a strong compression scheme, block sizes over 10,000 query/response pairs would seem to offer limited improvements.