Subject: [ippm] Ippm pdm

From: "Scheffenegger, Richard" <[email-masked]>
Date: Sun, 16 Nov 2014 14:31:05 +0000

Sorry for the sparse text and short (harsh) wording, writing this while on a plane...

I re-read the draft once more and these are the problema that i see with the pdm option as described and proposed.

1) service time can not be measured on the ip layer (example: nfs over rpc over tcp). At best, simple tcp may be captured for host stack response time (but never service/application response time). In any case, the proposed model breaks down for any communication models that are not p2p and e2e. In that case about 1/2 of the option is useless (always empty).

Please do not claim to be able to measure something here that is not feasible at that layer...

2) Timebase plus time scale plus delta is effectively a 2-times scaled floating number with an expoentent covering about 10^9*2^127. (draft describes a signed 8 bit value, not a signed 7bit). This seems excessive (also considering the effecitve precision of the clock insertion time).

3) Any studies as to how many unique 5-tuple sessions a busy web server (or, more applicable, oracle dnfs-enabled db server) serves over about 52 days?  I just wonder if keeping all that state plus an appropriate hash table is worth the possible gain. Imho, we are probably talking about millions of sessions (5tuples) for the dnfs oracle db server - each process aka query has a unique 5tuple; and probably tens of billions for a busy google or akamai frontend web server...

The proposed state is 5-tuple, plus 2 counters plus 2 unscaled timestamps, roughtly 128+128+16+16+8+16+16+128+128 bit per entry or 73 bytes before a hash, thus tables in the order of dozen MB to dozen GB...

4) Also, the draft forbids straight implementations alike ipid (single global counter across all packets, only outbound)

5) Is a 16 pkt cnts field not excessive? I guess the rationale here is an environment where traffic is sampled instead of captured in full, to get fewer unobserved counter wraps? However, no rationale is given for the counter size - there is more on the delta/time side

For many / most stuff and 2119 wording i miss a concise rationale.

6) Instead of describing in acute detail how to do a subtraction (sec 5) i would rather have a discussion about the expected implementation complexity, memory demands, effective scope (with precise nomenclature; as mentioned this only works for a simple tcp-ish point-to-point transport model; it probably already starts breaking down with mptcp, multicasts without doubt, and dtn protos (since they are alluded to) also.

7) Why would this proto have securit implications that go beyond those of a dest opt? Are you expecting all implementations to be buggy? 

Imho, i can see that something resembling ipid would be valueable (also with more bits, ie 16 if doing 2-tuple, or 32 bits if global counter).

I can also see that putting in some "absolute" time or time detala w/ defined clock base can benefit analysis. However that information imho should get derived from the used transport.

8) A discussion about when and where that clock is beig inserted (vm, paravirt driver, hypervisor, hardware,...) is fully absent, as is a discussion around the effective precision (making up some picosecond timer, in a vm, will at best have a jitter in the low millisecond range).

Best regards,
   Richard Scheffenegger

---

From: Nalini Elkins <[email-masked]>
Date: Sun, 16 Nov 2014 15:49:52 +0000 (UTC)

Richard,
Thanks for the comments and I will address them at length in the next 2 - 3 days when I have recovered from the flight & meetings!
One point I wanted to make though - from discussions that we have had with people, I realize that what was not made clear is that we assume an external correlation engine which will incorporate this data along with all the data that we can get from other layers (TCP seq no, mac addresses, ANYTHING!) to actually do diagnostics and response time calculations.
We want PDM to get some pieces of data that we cannot get about what the client actually knows about its experience.   I will write this up fully as well as addressing your comments.   I am so glad that you have written up your thoughts so that we can have the conversation.   Definitely we have a lot more work to do.
More to come ...

Nalini Elkins
Inside Products, Inc.
(831) 659-8360
www.insidethestack.com

      From: "Scheffenegger, Richard" <[email-masked]>
 To: "[email-masked]" <[email-masked]> 
Cc: "[email-masked]" <[email-masked]> 
 Sent: Sunday, November 16, 2014 6:31 AM
 Subject: [ippm] Ippm pdm

Best regards,
  Richard Scheffenegger
_______________________________________________
ippm mailing list
[email-masked]
https://www.ietf.org/mailman/listinfo/ippm

---

From: Nalini Elkins <[email-masked]>
Date: Tue, 18 Nov 2014 15:52:31 +0000 (UTC)

>Sorry for the sparse text and short (harsh) wording, writing this while on a plane...
No problem.  Thanks for the comments.

>1) service time can not be measured on the ip layer (example: nfs over rpc over tcp). At best, simple tcp may be captured for host stack response time (but never service/application >response time). In any case, the proposed model breaks down for any communication models that are not p2p and e2e. In that case about 1/2 of the option is useless (always empty).

>Please do not claim to be able to measure something here that is not feasible at that layer...
First, what we propose here gives us data that is missing to us from looking at both the upper layer protocol (tcp, udp) and layer 2.  (Ex. time from client to network - the last mile)
The proposed architecture is:
client (with PDM) -- network -- collector device (packet broker) -- network -- server
The output of the collector device then goes to a correlation engine which will use as many fields in all layers as possible.  For example, by looking at the MAC address, I can tell that this is a Canon device.   Then, I see multicast DNS being done.  So, I conclude that this may be a printer & then do performance analysis accordingly.  I am simplifying obviously to make a point.
The second point of the naming of "host response time" - I believe we called it "server delay".   Of course, it can be due to many things.  In any analysis, if one imagines there are 100 steps, PDM will only help us with step 1.   Other tools and mechanisms are needed for other steps.  
In the real world, I have seen literally days or weeks go by while people fight about step 1.
The point about data flows - I have made a list of data flow types that PDM needs to handle.  I am sure that I have packet traces with all such.   I will create a document which shows how PDM will be useful and what the contents will be for all such.  I did not do this before because if people do not understand what we are trying to do with the simplest case, no need to add complexity before it is needed.   These are the flows:
One way traffic flowTwo way traffic flowOne way traffic flow with keep-aliveTwo way traffic flow with keep-aliveMultiple send traffic flowMultiple receive traffic flow
Immediate ACK data flowDelayed ACK data flowProxied ACK data flow
If you have others, please let me know.

>2) Timebase plus time scale plus delta is effectively a 2-times scaled floating number with an expoentent covering about 10^9*2^127. (draft describes a signed 8 bit value, not a signed >7bit). This seems excessive (also considering the effecitve precision of the clock insertion time).
We will discuss this internally.  We wanted to be "future proof".  That is handle time which was in pico seconds to delays which may happen for DTN networks (traffic to Mars, for example.)

>3) Any studies as to how many unique 5-tuple sessions a busy web server (or, more applicable, oracle dnfs-enabled db server) serves over about 52 days?  I just wonder if keeping all >that state plus an appropriate hash table is worth the possible gain. Imho, we are probably talking about millions of sessions (5tuples) for the dnfs oracle db server - each process aka >query has a unique 5tuple; and probably tens of billions for a busy google or akamai frontend web server...

>The proposed state is 5-tuple, plus 2 counters plus 2 unscaled timestamps, roughtly 128+128+16+16+8+16+16+128+128 bit per entry or 73 bytes before a hash, thus tables in the >order of dozen MB to dozen GB...
This was a concern that we had also.  So, before proposing this header,  we talked  to two different OS vendors, both of whom service the largest enterprises in the world.  So, they are aware of scalability issues.   The people we talked to were the lead developers for the stack.  So, they know what is in their control blocks.   
They said that today, a TCP/IP stack in an OS maintains quite a bit of state.   A stack needs state to manage its connections.  They told us that what we were asking for, for TCP, may already be kept in control blocks.  
The issues are actually different for the stack - having to do with segmentation offload for TCP, etc.   But, I believe that Boeing has already done a Linux implementation of a Destination Options header.  So, we can discuss with them.
I am scheduling a discussion with another OS vendor in a week.  I am hoping to have one or more such vendors comment on the list with their comments.

>4) Also, the draft forbids straight implementations alike ipid (single global counter across all packets, only outbound)
Many OS vendors implement IPID by 5-tuple today.  We have talked with one who does not & they said that it would not be a problem for them.

>5) Is a 16 pkt cnts field not excessive? I guess the rationale here is an environment where traffic is sampled instead of captured in full, to get fewer unobserved counter wraps? >However, no rationale is given for the counter size - there is more on the delta/time side

>For many / most stuff and 2119 wording i miss a concise rationale.
Thanks.  Will provide more wording.

>6) Instead of describing in acute detail how to do a subtraction (sec 5) i would rather have a discussion about the expected implementation complexity, memory demands, effective >scope (with precise nomenclature; as mentioned this only works for a simple tcp-ish point-to-point transport model; it probably already starts breaking down with mptcp, multicasts >without doubt, and dtn protos (since they are alluded to) also.
See answer to question 1.  Will provide further discussion.

>7) Why would this proto have securit implications that go beyond those of a dest opt? Are you expecting all implementations to be buggy? 
I am not understanding this comment.  Can you expand?

>I can also see that putting in some "absolute" time or time detala w/ defined clock base can benefit analysis. However that information imho should get derived from the used transport.
We initially proposed an "absolute" time.   Many people felt that time synchronization would be our downfall.  We actually have a proposal which uses absolute time.  I can revive that for the group.  We ourselves prefer that implementation.
>8) A discussion about when and where that clock is beig inserted (vm, paravirt driver, hypervisor, hardware,...) is fully absent, as is a discussion around the effective precision (making >up some picosecond timer, in a vm, will at best have a jitter in the low millisecond range).
Thanks.  Will provide this.

Best regards,
  Richard Scheffenegger