Subject: Proposal for PSAMP-PROTO section 6.5.2.6

From: Andrew Johnson <[email-masked]>
Date: Fri, 03 Mar 2006 20:30:25 +0000

Hello all

Below is the proposed text for the PSAMP protocol section 6.5.2.6
(Hash-Based Filtering) and for the changes to the Basic Packet
Report to include the result of a Packet Digest Function.

Things to note:
 - The input to the hash function is mandated and fixed.
 - CRC, IPSX and BOB MAY be used for filtering or packet digest.
 - To ensure interoperability certain configurable ranges are
   mandated.  Are these ranges appropriate?
 - To stop someone has snooped the hash configuration from shaping
   their traffic to manipulate detection the initialisation value
   is optional.  Is this sufficient?  Does it only work with BOB?

Suggested change to basic packet report text:

===================================================================
For each selected packet, the Packet Report MUST contain the
following information:
- ...
- The hash value (digestHashValue) generated by the digest hash
  function.  If there are no digest functions in the selection
  sequence then no element needs to be sent.  If there are more than
  one digest function then each hash value must be included in
  the same order as they appear in the selection sequence.
===================================================================

Potentially we can add this to the example:

===================================================================
IPFIX Template Record:

0                   1                   2                   3
0 1 2 3 4 5 6 7 8 9 0 1 2 3 4 5 6 7 8 9 0 1 2 3 4 5 6 7 8 9 0 1
+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+
|           Set ID = 2          |         Length = 20           |
+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+
|        Template ID = 260      |        Field Count = 2        |
+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+
|       selectionPath = 321     |        Field Length = 4       |
+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+
|      digestHashValue = 326    |        Field Length = 4       |
+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+
|  ipHeaderPacketSection = 313  |        Field Length = 12      |
+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+

The associated IPFIX Data Record:

0                   1                   2                   3
0 1 2 3 4 5 6 7 8 9 0 1 2 3 4 5 6 7 8 9 0 1 2 3 4 5 6 7 8 9 0 1
+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+
|           Set ID = 260        |           Length = 24         |
+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+
|                               9                               |
+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+
|                         0x9123 0613                           |
+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+
|                         0x4500 005B                           |
+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+
|                         0xA174 0000                           |
+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+
|                         0xFF11 832E                           |
+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+

       Figure D: Example of a Basic Packet Report
===================================================================

Note: this means that any digest hash function must take the same
parameters as a selection hash function.  I think this is currently
the best option for interoperability.

Secondly we will need a report to communicate the configuration
of the hash-based selector to the Collecting Process.

===================================================================
6.5.2.6 Hash-Based Filtering

In hash based selection a hash function is run on IPv4 traffic
the following fields MUST be used as input to that hash function:
  - IP identification field
  - Flags field
  - Fragment offset
  - Source IP address
  - Destination IP address
  - A number of bytes from the IP payload.  The number of bytes
    and starting offset MUST be configurable if possible.

For the bytes taken from the IP payload, IPSX has a fixed offset
of 0 bytes and a fixed size of 8 bytes.  The number and offset of
payload bytes in the BOB function MUST be configurable.  If any
of the configured set of bytes from the IP payload are unavailable
then 0 MUST be used, which may result in a different value than
if the hash function is run on a subset of the input.

The minimum configuration ranges MUST be as follows:
  Number of bytes:  from 8 to 32
  Offset:           from 0 to 64

If the selected payload bytes are not available and the hash function
can take a variable sized input then the hash function MUST be run
with the information which is available and a shorter size.  Passing
0 as a substitute for missing payload bytes is only acceptable if
the hash function takes a fixed size as is the case with IPSX.

If the hash function can take a initialisation value then this
value MUST be configurable.

A hash-based selection function MAY be configurable as a digest
function.  Any selection process which is configured as a digest
function MUST have the output value included in the basic packet
report for any selected packet.

Each hash function used as a hash-based selector requires it's own
value for the selectorAlgorithm. Currently we have BOB (6), IPSX (7)
and CRC (8) defined and any MAY be used for either either Filtering
or creating a Packet Digest.  Only BOB is recommended though and
SHOULD be used.

The REQUIRED algorithm specific Information Elements in case of hash
based selection are:

hashIPPayloadOffset   - The configured or set payload offset
hashIPPayloadSize     - The configured or set payload size
hashOutputRangeMin    - One or more values for the beginning of
                        each potential output range.
hashOutputRangeMax    - One or more values for the end of each
                        potential output range.
hashSelectedRangeMin  - One or more values for the beginning of
                        each selected range.
hashSelectedRangeMax  - One or more values for the end of each
                        selected range.
hashDigestOutput      - A boolean value, TRUE if the output from
                        this selector has been configured to be
                        included in the packet report as a packet
                        digest.

NOTE: If more than one selection or output range needs to be sent
then the minimum and maximum elements may be repeated as needed.
These MUST make one or more non-overlapping ranges.  The elements
SHOULD be sent as pairs of minimum and maximum in ascending order,
however if they are sent out of order then there will only be one
way to interpret the ranges to produce a non-overlapping range and
the Collecting Process MUST be prepared to accept and decode this.

The following algorithm specific Information Element MAY be sent,
but is optional for security considerations:
hashInitialiserValue  - The initialiser value to the hash function.

Example of a hash based filter Selector, whose configuration is:
Hash Function           = BOB
Hash IP Payload Offset  = 0
Hash IP Payload Size    = 16
Hash Initialiser Value  = 0x9A3F9A3F
Hash Output Range       = 0 to 0xFFFFFFFF
Hash Selected Range     = 100 to 200 and 400 to 500

IPFIX Options Template Record:

0                   1                   2                   3
0 1 2 3 4 5 6 7 8 9 0 1 2 3 4 5 6 7 8 9 0 1 2 3 4 5 6 7 8 9 0 1
+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+
|             Set ID =  3       |          Length = 50          |
+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+
|        Template ID = 269      |       Field Count = 8         |
+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+
|     Scope Field Count = 1     |0|     selectorId = 300        |
+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+
|      Scope 1 Length = 4       |0|   selectorAlgorithm = 302   |
+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+
|       Field Length = 1        |0|  hashIPpayloadOffset = 327  |
+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+
|       Field Length = 4        |0|   hashIPpayloadSize = 328   |
+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+
|       Field Length = 4        |0|  hashInitialiserValue = 329 |
+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+
|       Field Length = 4        |0|   hashOutputRangeMin = 330  |
+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+
|       Field Length = 4        |0|   hashOutputRangeMax = 331  |
+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+
|       Field Length = 4        |0|  hashSeletionRangeMin = 332 |
+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+
|       Field Length = 4        |0|  hashSeletionRangeMax = 333 |
+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+
|       Field Length = 4        |0|  hashSeletionRangeMin = 332 |
+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+
|       Field Length = 4        |0|  hashSeletionRangeMax = 333 |
+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+
|       Field Length = 4        |
+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+

Associated IPFIX Data Record:

0                   1                   2                   3
0 1 2 3 4 5 6 7 8 9 0 1 2 3 4 5 6 7 8 9 0 1 2 3 4 5 6 7 8 9 0 1
+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+
|           Set ID = 266        |        Length = 45            |
+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+
|                              22                               |
+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+
|       6       |                            ...                |
+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+
| ...   0       |                            ...                |
+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+
| ...  16       |                      0x9A3F9A ...             |
+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+
| ...  3F       |                            ...                |
+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+
| ...   0       |                      0xFFFFFF ...             |
+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+
| ...  FF       |                        ... 100                |
+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+
|      ...      |                        ... 200                |
+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+
|      ...      |                        ... 400                |
+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+
|      ...      |                        ... 500                |
+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+
|      ...      |
+-+-+-+-+-+-+-+-+

  Figure K: Example of the Selector Report Interpretation,
            for Hash Based Filtering

Notes:
* A selectorAlgorithm value of 6 represents hash-based Filtering
 using the BOB algorithm.

===================================================================

--
to unsubscribe send a message to [email-masked] with
the word 'unsubscribe' in a single line as the message text body.
archive: <http://ops.ietf.org/lists/psamp/>

---

From: Youki Kadobayashi <[email-masked]>
Date: Thu, 23 Mar 2006 06:53:18 +0900

This is a minor point..

> then the minimum and maximum elements may be repeated as needed.

This part talks about both output ranges and selection ranges.

> These MUST make one or more non-overlapping ranges.

Perhaps you are talking about selection ranges here,
since multiple digest hash functions can take overlapping
output ranges...

 > The elements

Thanks,
	Youki Kadobayashi
	WIDE Project

---

From: Andrew Johnson <[email-masked]>
Date: Thu, 23 Mar 2006 06:32:36 +0000

> since multiple digest hash functions can take overlapping

No, I am talking about either selection or output range.  This is
for communicating the ranged of a single digest hash function, so
overlapping ranges do not make sense.  Furthermore, an overlapping
range could just be communicated by sending the larger range, e.g.
1 to 10 plus 5 to 15 => 1 to 15.

Although it is recommended that the range I.E.s are sent in a
sensible order this is not mandated.  As such, the min and max
values could be sent in any order (and a collector MUST cope
with this).  This works OK because there is only one way to
interpret the list as long as there can be no overlaps.

regards

Andrew

---

From: Andrew Johnson <[email-masked]>
Date: Fri, 24 Mar 2006 00:04:59 +0000

Hi all

There is a contradiction in the proposed text that I would like to
correct.

Andrew Johnson wrote:
[SNIP]
> 6.5.2.6 Hash-Based Filtering
[SNIP]
> if the hash function is run on a subset of the input.

Here, for a hash function with a variable length input, it says that
you must pass in 0 for the missing bytes.  This disagrees with the
text below and I would like to simply remove the last sentence.

> the hash function takes a fixed size as is the case with IPSX.

The above text is quite clear, and contrary to the previous original
paragraph.

regards

Andrew