Subject: [ippm] Ippm pdm

From: Nalini Elkins <[email-masked]>
Date: Tue, 18 Nov 2014 20:49:35 +0000 (UTC)

Richard,

Thank you for your comments. 
>1) service time […]
>Please do not claim to be able to measure something here that is not feasible at that layer...

-First, what we propose here gives us data that is missing to us from looking at both the upper layer protocol (tcp, udp) and layer 2.  (Ex. time from client to network - the last ---mile)

>Relative Timing can be extracted by the observation point without in-packet information. Loss potentially not, and when doing sampling captures, it becomes problematic as well… In >short, I see some relevance for a delta-since-last-sent-segment (but with not quite as much emphasis as the packet counter piece), much less so for the delta-since-last-received, and >packet-number-last-received. I guess my main bone remains the unclear relevance of the receiver side timing/packet number counters, in anything but TCP really. 

In large enterprises, protocols embedded in UDP are widely used.  In fact, it was one of these use cases that originally started our exploration on this path.  TCP is much too limited.  We need to be able to handle any upper layer protocol.  You may want to look at Fred Templin's presentation of the IPv6 Destination Option that he proposed.

https://tools.ietf.org/html/draft-skeen-6man-ipv6geo-01

His slides are here.  He makes the point that IP is the commonality between many disparate types of functions.  

http://www.ietf.org/proceedings/91/slides/slides-91-6man-8.pdf (page 5)

-The proposed architecture is:

-client (with PDM) -- network -- collector device (packet broker) -- network -- server

-The output of the collector device then goes to a correlation engine which will use as many fields in all layers as possible.  For example, by looking at the MAC address, I can tell -that this is a Canon device.   Then, I see multicast DNS being done.  So, I conclude that this may be a printer & then do performance analysis accordingly.  I am simplifying obviously -to make a point.

-The second point of the naming of "host response time" - I believe we called it "server delay".   Of course, it can be due to many things.  In any analysis, if one imagines there are -100 steps, PDM will only help us with step 1.   Other tools and mechanisms are needed for other steps.  

>I believe in your talk you mentioned „service time“. Which is probably ok, if it is clear, which service (ie. TCP in the OS) you refer to. However, the common use of “service time” >is not really in the transport layer, but more like session to application layer. And the delta times of a 5-tuple response to a 5-tuple packet are unlikely to truly match here.

I hope that it will be useful, actionable information.  

-In the real world, I have seen literally days or weeks go by while people fight about step 1.

-The point about data flows - I have made a list of data flow types that PDM needs to handle.  I am sure that I have packet traces with all such.   I will create a document which shows -how PDM will be useful and what the contents will be for all such.  I did not do this before because if people do not understand what we are trying to do with the simplest case, no --need to add complexity before it is needed.   These are the flows:

-One way traffic flow
-Two way traffic flow
-One way traffic flow with keep-alive
-Two way traffic flow with keep-alive
-Multiple send traffic flow
-Multiple receive traffic flow

-Immediate ACK data flow
-Delayed ACK data flow
-Proxied ACK data flow

-If you have others, please let me know.

>In this list, what about SAP / IP Multicast services; a number of ip hosts send to a well-known IPMC dst; a client sends a response to a (different) IPMC address, announcing it wants to >receive a IPMC stream with an IPMC address extracted from the SAP packets

>If a majority of flows effectively look unidirectional on the L5 tuple, half of the bytes in the option go wasted, not?

>Why is a two-way w/ keepalive different from a two-way w/o keepalive? I assume that all the ACK data flows are a subset of the two-way, or is that different? I don’t quite understand the differences in that list, as I only see 2 relevant entries (the first two), and the others being an ensemble of a number of those, at best…

The differences are for how the correlation engine must calculate what the numbers mean.  Interpreting what a packet means which is an ACK with a PDM is quite different from getting a packet with data in it.

>2) Timebase plus time scale plus delta is effectively a 2-times scaled floating number with an expoentent covering about 10^9*2^127. 
>(draft describes a signed 8 bit value, not a signed 7bit). This seems excessive (also considering the effecitve precision of the clock insertion time).
-We will discuss this internally.  We wanted to be "future proof".  That is handle time which was in pico seconds to delays which may happen for DTN networks (traffic to Mars, for --example.)

>Mars is max. 15 light minutes away ;) Even Pluto, though no longer regarded a planet, is only around 8-9 light hours. If the sender is stateful, and has to maintain a 5-tuple for the >maximum time of 50+ days, this sounds excessive.

>As you have read the timestamps drafts by Brian, Mirja and myself, you should probably be aware of the point that multiple representations can be choosen for the same time delta; I >may have missed where you describe how to select an appropriate representation (IEEE floats get one bit free by stipulating the leftmost bit of the mantissa is always one…)

Will look at this.

>3) Any studies as to how many unique 5-tuple sessions a busy web server (or, more applicable, oracle dnfs-enabled db server) serves over about 52 days?  I just wonder if keeping all >that state plus an appropriate hash table is worth the possible gain. Imho, we are probably talking about millions of sessions (5tuples) for the dnfs oracle db server - each process aka >query has a unique 5tuple; and probably tens of billions for a busy google or akamai frontend web server...

>The proposed state is 5-tuple, plus 2 counters plus 2 unscaled timestamps, roughtly 128+128+16+16+8+16+16+128+128 bit per entry or 73 bytes before a hash, thus tables in the >order >of dozen MB to dozen GB...

-This was a concern that we had also.  So, before proposing this header,  we talked  to two different OS vendors, both of whom service the largest enterprises in the world.  So, they -are aware of scalability issues.   The people we talked to were the lead developers for the stack.  So, they know what is in their control blocks.  

-Unlike a transport protocol control block, you don’t have any clue when a session (L5 tuple) ends, on L2 – thus you are forced to keep that information around for a maximum time (in -case the transport decides to send a follow up packet). If the transport decided to re-use a port-point towards a host, forming a new L4 session, it will have to look as a --continuation of the previous session on L2 here…

-To the same point, TCP is not known to keep state for very much longer than 2*MSL;  And very busy frontend webservers run stateless for the majority of a session lifetime (initial -SYN till ACK, then a IW burst of data, followed by a FIN, the last couple packets taking about as long as the first).

-In your talk, you alluded not only to OS Stack vendors (Linux having the largest share, which is not a “OS vendor” per se) but NIC / HW manufacturers as well… 

-Finally, if I understand this correct, the IPv6 DestOpt seems to be a workaround for a TCP option (or ensemble of options, like TS, SACK, [Echo])?

What we want are:

- metrics that can be commonly known and interpreted no matter what the protocol
- something that network operations and diagnosticians can turn on / off easily and interpret easily

I don't know that it is a "work around".  Many protocols seem to be re-inventing what should have been there to start with.

-They said that today, a TCP/IP stack in an OS maintains quite a bit of state.   A stack needs state to manage its connections.  They told us that what we were asking for, for TCP, may -already be kept in control blocks.  

>Yes, but again, TCP state, while many dozen or hundred bytes per session, is usually recycled after a few seconds, not 50+ days... I guess you can tie this into the TCP transport. >But then the question arises, why not make a generic TCP option, which would work not only for IPv6, but IPv4 as well? And as I tried to convey earlier, much of this is not useful >for anything but TCP anyway; finally, TSopt is already there and could easily convey this information (ie. run an “absolute” clock on the 24 MSB bits, and the delta from last->received in the 8 LSB; or use the upcoming TCP (new) Echo option which I’m co-authoring.

I do not agree with you in the least about TCP only.  And in fact, the IP sequence number can be quite helpful for TCP retransmit issues.  It is hard to diagnose a layer from within itself.  It is good to have another viewpoint.

Your point about the 50+ days to maintain state is well taken.  I need to think about a timeout.  I do not know about the "recycled after a few seconds" business though.  I have done plenty of stack diagnostics where the control block & actually buffers stick around for DAYS until the session ends.   In fact, this is a known problem in some stacks.

-The issues are actually different for the stack - having to do with segmentation offload for TCP, etc.   But, I believe that Boeing has already done a Linux implementation of a -Destination Options header.  So, we can discuss with them.

-I am scheduling a discussion with another OS vendor in a week.  I am hoping to have one or more such vendors comment on the list with their comments.

>4) Also, the draft forbids straight implementations alike ipid (single global counter across all packets, only outbound)

-Many OS vendors implement IPID by 5-tuple today.  We have talked with one who does not & they said that it would not be a problem for them.

>There is no IPv6 IPID;

I know.  In fact, it was our discovery that this important field which in IPv4, for quite a few stacks had the properties of being sequential, was missing (or rather moved to the Fragmentation Extension Header) that led us to where we are today.   We first brought up this issue in v6Ops about 3 years ago.

> and yes, it’s completely a choice by the IP stack by which means the IPID is filled (static, global counter, flow counter, random…).
>I can only repeat, I see the most value in the packet counter, which allows to uniquely identify retransmissions and other interesting cases.  However, your draft uses 2119 language to forbid all but one >method as to how that field is to be set.

Yes.   We need to know what we are getting.   It CANNOT be a choice by the stack.

>7) Why would this proto have securit implications that go beyond those of a dest opt? Are you expecting all implementations to be buggy? 

I am not understanding this comment.  Can you expand?

”6  Security Considerations

   TBD.   It is conceivable that in allowing this Destination Option
   through a firewall, that other malicious traffic may be allowed
   through.
“

>Besides, the real security implication (apart from buggy firewall implementations, and blue-eyed firewall admins) of YOUR proposal is the exposure of host-specific, and flow-specific >information; So you can do all these side-band analysis of (user) behaviour with a much higher fidelity signal than if you’d otherwise passively monitor traffic at some random point, >as everything is in the clear. Your counters can expose internal state of both sender and receiver (ie. knowledge that a cetain critical day-zero apparently wasn’t fixed here, as the >box was not rebooted since release of the patch, for example…) For TCP, some aspects (ie. information leaked in TCP timestamps) are listed here https://tools.ietf.org/html/draft-ietf-tcpm-tcp-security and you may want to analyze the information you expose with these scenarios in mind. (The timing analysis vector is not described therein though); What you do >for the purpose of troubleshooting (timing analysis) also prevents any meaningful protection, unless the data is scrambled with a shared secret perhaps (shared between analyst <> >sender, analyst<>receiver; the receiver can pretty much do what you propose by treating the “PSN this” as opaque blob of data; the rest is local data anyway.

>Note that the IAB requires all new protocols to include discussions around such points, due to the new threat model of the internet. I’m by no means a security expert, but guess that >something simple as described above addresses security, (timing) confidentiality and integrity issues.

Will look at this & rework the draft.

>Imho, i can see that something resembling ipid would be valueable (also with more bits, ie 16 if doing 2-tuple, or 32 bits if global counter).

>I can also see that putting in some "absolute" time or time detala w/ defined clock base can benefit analysis. However that information imho should get derived from the used transport.

We initially proposed an "absolute" time.   Many people felt that time synchronization would be our downfall.  We actually have a proposal which uses absolute time.  I can revive that for the group.  We ourselves prefer that implementation.

I know about the problems of absolute time and synchonization to an extent that I don’t like JRelative time, and an idea of the attainable precision (see my comments earlier on about different representations of the same value) is better.

Best regards,
  Richard