Subject: Deering's comments on flow ID management

From: Craig Partridge <[email-masked]>
Date: Tue, 29 Nov 94 11:11:47 -0800

Hi folks:
    
    Steve's comments on an earlier draft and my note to these lists crossed
in the ether this morning.  Steve suggested I forwarded his comments on to
everyone.

Craig

E-mail: [email-masked] or [email-masked]

------- Forwarded Message

To: Craig Partridge <[email-masked]>
Cc: [email-masked]
Subject: Re: rough draft of flow id management memo 
Date: 	Tue, 29 Nov 1994 09:13:34 PST
Sender: Steve Deering <[email-masked]>
From: Steve Deering <[email-masked]>

Craig,

Sorry this took so long -- so much to do and so little time and energy.

> Please give me your comments soon, as I'd like to send this to the INT SERV
> next week for discussion at IETF.  (I assume I should also hit the
> E2E-Interest list -- any others?).

Please also send it to the ipng and sdrp lists, and maybe to Noel's flows
list.

> I'm treating this as ideas from the End-to-End Research Group and propose
> to publish it as an informational RFC. (The RG so rarely produces things
> I thought it would be nice, and is certainly consistent with the idea of
> publishing white papers sent to the IPng process).

Sounds fine to me.  Suggest you get it out as an Internet Draft first,
in case there is delay in the RFC-ification process.

>    The specification further requires that all datagrams with the same
>    (non-zero) Flow ID must have the same Destination Address, Hop
>    Options header, and Routing Header contents.

Insert "and Source Address" before "must have".
Change "Hop Options" to "Hop-by-Hop Options".

> What Does a Router Do With Flow IDs for Which It Has No State?
> 
>    If a datagram with a non-zero Flow ID arrives at a router and the
>    router discovers it has no state information for that Flow ID, what
>    is the correct thing for the router to do?
> 
>    There is an initial instinct to assume that this scenario only occurs
>    in the case of an error,

That would not be one's initial instinct if one read the spec.  The spec
says that routers are free to ignore the flow ID if they don't implement
support for flows; clearly in that case packets may arrive with non-zero
flow IDs and the absence of flow state would not be an error.  The spec
also talks about flows carrying their set-up information in options
within the flow's data packets themselves, in which case it would be quite
normal and non-erroneous for a packet with a non-zero flow ID to arrive and
find no state yet (that state to be created when the packet's options are
processed).

>    In any case, it is clear that usual IPv4 error handling, namely to
>    drop the datagram and send an ICMP message, is inappropriate.  We
>    expect a large number of flows to be multicast, and it is easy to
>    envision scenarios in which the flow's sender could be innundated
>    with ICMP error messages if some portion of the internet unexpectedly
>    starts receiving the flow's datagrams.

Since the previous paragraph pointed out that it is not erroneous to be
missing state, there's no need to start talking about "IPv4 error handling"
(which also makes one wonder why you are talking about IPv4 at all).

Your multicast scenario is incorrect because there are overriding rules
about not sending ICMP error messages in response to multicasts, which
prevent the behavior you describe.

"innundated" -> "inundated"

>    In summary, the group's view is that the default rule should be that
>    if a router receives a datagram with an unknown Flow ID, it treats
>    the datagram as if the Flow ID and traffic class fields are zero.  As
>    part of forwarding, the router will examine any options...

More precisely, the router will examine any hop-by-hop options present in
the packet (and, possibly, additional extension headers if the packet were
explicitly addressed to the router).

>                                             The options could include
>    simply the information that the datagram is to be dropped if the Flow
>    ID is unknown or could contain a brief summary of the flow state the
>    router should have.

A "brief summary"?  What use would that be?  If the option includes more than
just error-handling instructions, shouldn't it include all the information
necessary to (re)establish the state?

>    State Flow IDs can happen a number of ways, ...

"State" -> "Stale"

>    ...flow was created.  Or the deletion message may be discarded before
>    reaching all routers.

"discarded" -> "lost"

>                                           ...The point here is that we
>    cannot expect the source (or, for the same reasons, a third party) to
>    clear out old Flow IDs.  Rather routers will have to find some
>    mechanism to flush Flow IDs themselves.

Insert "always" before "to clear out".
Insert a comma after "Rather".

>    The obvious mechanism is to use a timer.  Routers should discard Flow
>    IDs whose state has not been refreshed within some period of time
>    (say 2 minutes).  At the same time, a source that crashes must
>    observe a quiet time,...

Note that this applies only to sources that are unable to remember their
flows across crashes, e.g., in stable storage.

>    Exactly how the Flow ID and its state should be refreshed needs some
>    study.  There are two obvious options.  The source could periodically
>    send out a special refresh message (such as an RSVP Path message) to
>    explicitly refresh the Flow ID and its state.  Or, the router could
>    treat every datagram that carries the Flow ID as an implicit refresh.

There is at least one more alternative: explicit refreshing info could be
piggybacked (i.e., carried as an option) on some or all of the flow's data
packets.

>                                          ...And real-time flows must
>    obviously always have a Flow ID, since flows are the reason Flow IDs
>    were created.

Wrong.  *Explicitly set up flows* or *flows with reserved resources* must
obviously always have a flow ID.  We may well decide to continue handling
some/most real-time traffic as unreserved, non-setup datagrams, as we do
now.  In other words flowness and real-timeness are orthogonal.

>                                                          ...Some people
>    want TCP connections to use Flow IDs, others do not.

Insert "all" before "TCP connections".

>    The argument in favor of using Flow IDs on individual TCP connections
>    is that even if the source does not request special service, a net-
>    work provider's routers may be able to recognize a large amount of
>    traffic and use the Flow ID field to establish a special route that
>    gives the TCP connection more bandwidth or lower delay.

This seems a very poorly thought-out argument to me.  Since flow IDs are
random, and therefore unpredictable by the providers, and since TCP
connections come and go quite quickly (especially since Web traffic started
to dominate), it seems impractical and of marginal value to do what you
suggest here.  More likely would be re-routing all TCP connections between
a pair of nodes (or clusters) -- the flow ID wouldn't help at all for that.

The only credible argument I have heard for labelling every TCP connection
with a unique flow ID is for efficient demux at the receiver.  And I don't
know how much of a performance improvement that would buy, given that folks
have already implemented zero-copy TCPs under IPv4, which does not have
flow IDs.

It has also been suggested that labelling every TCP connection with a
unique flow ID would somehow facilitate header compression, but as you
saw in the e2e meeting, I go ballistic at that suggestion.  :-)

>             Putting Flow IDs in each datagram changes the cache into a
>    Flow ID cache, in which there's a cache entry for every TCP connec-
>    tion.  So there's a potential for cache explosion.

The amount of explosion will depend on the "working set" of active flows,
presumably.  That *might* not be so bad -- should look at the various recent
TCP trafric sttudies.

> 2.   Flow IDs should expire if not refreshed within some number of
>      minutes (the exact number needs some more consideration) and flow
>      sources should observe a quiet time equal to this interval after
>      rebooting to avoid confusing Flow IDs.

Again, include caveat about sources that can remember flows across reboots.

Did you get a chance to talk to Van about the costs/benefits of flow IDs
for all non-reserved TCP connections?

Steve

------- End of Forwarded Message

---

From: Zubin Dittia <[email-masked]>
Date: Wed, 30 Nov 1994 03:39:12 -0600 (CST)

> have already implemented zero-copy TCPs under IPv4, which does not have

Setting the Flow Label to zero for packets not belonging to a
flow seems wasteful; it would be nice if there were some way to
differentiate between "network" flows (i.e., flows that need to be
regarded as flows by both routers and end-points), and "end-flows"
(i.e., flows that are regarded as flows only by the end-points, and
not by the routers).  If this could be done (perhaps by means of a
bit in either the Flow ID or the TClass fields), then we can get
the advantage of efficient demuxing at the receiver, without the
drawback of cache explosion in the routers.  That would argue for
letting every TCP connection have its own Flow ID.

Regarding the usefulness of interpreting flow IDs in the end-host's
network adapter, the potential is always there, and goes beyond
just achieving zero-copies.  For example, the flow ID field could be
used by a network adapter to know where to look in the packet for
specific information that can be interpreted by the adapter.  This
would allow, for example, the source of a flow to control when an
interrupt is issued by the network adapter on the destination host;
such a feature can be immensely useful in reducing the number of
interrupts that are issued.  Another example is with distributed
shared memory, where the flow ID could be used to find where the data
resides in the packet, and to DMA that data directly to its destination
address in the host's memory.

-Zubin.