Subject: [tcpm] Re: [Tsvwg] Re: [e2e] Are you interested in TOEs and related issues

From: "Alexandre L. Grojsgold" <[email-masked]>
Date: Sun, 7 Mar 2004 19:06:56 -0300 (E. South America Standard Time)

>
> The problem is that in today's routers the buffer sizes are typically
> accounted in packets, not in bytes. So looking at your example, and
> supposing that the line card buffer is limited to 100 packets, a burst
> of small frames will instantly consume 13% of available buffer slots,
> while jumbos will only use 2%. This is of course no problem if the line
> card can transmit all those frames instantly, but what if it cannot?

So what?

Small packets get more buffer frames, but are also more rapidly sent.
The gueue grows fats, but shrinks fast too. It means that a burst of small
packets will not increase the packet loss, no mather the length of the
buffer (in packets) .

Ok, if the router gets a large amount of packets, representing more bytes
per second than the capacity of the output line, packets will be lost. But
it is a function of the amount of data conveyed by the packets, not of
their individual length.

>
>
> >
> > >> BTW, I asked a few transport folks in Minneapolis IETF about how
> > >> "evil" is traffic burst in today's enviroment, but did not get any
> > >> concrete answer. Perhaps this topic should be discussed in tsvwg
> > >> or tcpm.
> > >
> > >Because queues in todays routers have finite maximum lengths, and
> > > this model is unlikely to change in the forseeable future,
> > > excessive traffic bursts will be more likely subject to drop-tail
> > > policing than other kinds of more smoothly shaped traffic. More
> > > than that, the bursty traffic will not only have less chance of
> > > reaching its target with all fragments in place, but it will also
> > > most probably do much harm to

I think many people tend to thing about IP routers like old circuit
switched protocols switches, like X.25.

In circuit switched networks, flow control is generaly implemented in each
link, each pair of switches exchanging acks and naks, trying to keep data
at the right pace.

With IP, the flow control is end-to-end, performed by TCP. If a router
queues too many packets, it will break TCP stability. It is not an issue
of memory or memory price.

alg.

_______________________________________________
tcpm mailing list
[email-masked]
https://www1.ietf.org/mailman/listinfo/tcpm

---

From: Marko Zec <[email-masked]>
Date: Mon, 8 Mar 2004 16:46:47 +0100

On Sunday 07 March 2004 23:06, Alexandre L. Grojsgold wrote:
> > The problem is that in today's routers the buffer sizes are
> > typically accounted in packets, not in bytes. So looking at your
> > example, and supposing that the line card buffer is limited to 100
> > packets, a burst of small frames will instantly consume 13% of
> > available buffer slots, while jumbos will only use 2%. This is of
> > course no problem if the line card can transmit all those frames
> > instantly, but what if it cannot?
>
> So what?
>
> The gueue grows fats, but shrinks fast too. It means that a burst of
> small packets will not increase the packet loss, no mather the length
> of the buffer (in packets).

This theory is unfortunately wrong, because it ignores the fact that 
when congestions do occur, traffic flows consisting of small packets 
(regardless whether they are bursty or smooth) will use more buffer 
slots then the flows carrying the larger ones, leaving less remaining 
buffer slots available for queuing further packets / bursts.

> Ok, if the router gets a large amount of packets, representing more
> bytes per second than the capacity of the output line, packets will
> be lost. But it is a function of the amount of data conveyed by the
> packets, not of their individual length.

Can you define the function you are talking about more precisely? What 
does it represent (I assume packet loss ratio)? Which argument(s) does 
it take? If this function is as simple as it might sound from your 
statement, why don't you share it with us?

>
> > > >> BTW, I asked a few transport folks in Minneapolis IETF about
> > > >> how "evil" is traffic burst in today's enviroment, but did not
> > > >> get any concrete answer. Perhaps this topic should be
> > > >> discussed in tsvwg or tcpm.
> > > >
> > > >Because queues in todays routers have finite maximum lengths,
> > > > and this model is unlikely to change in the forseeable future,
> > > > excessive traffic bursts will be more likely subject to
> > > > drop-tail policing than other kinds of more smoothly shaped
> > > > traffic. More than that, the bursty traffic will not only have
> > > > less chance of reaching its target with all fragments in place,
> > > > but it will also most probably do much harm to
>
> In circuit switched networks, flow control is generaly implemented in
> each link, each pair of switches exchanging acks and naks, trying to
> keep data at the right pace.
>
> With IP, the flow control is end-to-end, performed by TCP. If a
> router queues too many packets, it will break TCP stability. It is
> not an issue of memory or memory price.

Agreed. And that's precisely the reason why bursty traffic is bad - 
since routers are generally tuned to queue only as few packets as 
feasible for normal TCP operation, excessive traffic bursts will have 
more chance of being tail-dropped at router queues.

Marko

---

From: "H.K. Jerry Chu" <[email-masked]>
Date: Sat, 6 Mar 2004 08:48:14 -0800 (PST)

...
<snip>

>> It's mainly the TCP receive window, not the PMTU that dictates how
>> much buffering will be needed. Routers that do cut-through need not
>> buffer the whole pkt. More significant is the drop of the DRAM price
>> and increase of the DRAM size making memory a non-issue in many cases
>> today.
>
>
>If the DRAM price alone were the only factor preventing vendors from 
>implementing unlimited packet queues, we would already have seen tons 
>of such devices on the market over the past 10 years or so. Excessive 
>queuing delays on routers are bad, especially for high-speed traffic 
>managed by TCP-like congestion control schemes. The queue lengths on 
>routers have always been and will always present a tradeoff between a 
>smallest usable window for handling traffic bursts, and the desire of 
>keeping the queueing delays as low as possible on the other hand.

Assuming pkt switching speed has scaled up too, I don't see why today's
much faster routers with a larger max queue length will cause the
average queuing delay to be any longer than routers 10 years ago.

A perhaps oversimplified comparison can be made between a conforming
traffic using 9K jumbo frames vs a non-conforming, traffic burst of 13
back-to-back regular Ethernet frames. They both require the same amount
of memory bandwidth in the forwarding path (2*9K ~= 13*1500). If all
the forwarding optimization in today's routers has made the switching
speed of 13 consecutive Ethernet "cells" from the same flow similar to
2 back-to-back jumbo frames, why would the former be any more evil than
the latter?

>
>
>>
>> Over the past decade many components involved in providing high-speed
>> networking have scaled up an order of magnitude. This including link
>> bandwidth, CPU speed, I/O bus, memory size..., but not the Ethernet
>> MTU and certain TCP parameters (such as the every-other-pkt acking
>> policy). This is really hurting the throughput performance of the
>> hosts. IMHO the amount of burstiness by TCP over WAN should be
>> allowed to scale up an order of magnitude too. If stretch ACKs are
>> fully adopted into TCP algorithm (see rfc2525 for a number of issues
>> with stretch acks), one can use LSO on the transmit side, and
>> per-flow pkt coalescing on the receive side to provide effectively a
>> simple, stateless AAL5 layer for the Ethernet "cells" without
>> requiring jumbo frames or complex TOE engine.
>>
>> concrete answer. Perhaps this topic should be discussed in tsvwg or
>> tcpm.
>
>
>Because queues in todays routers have finite maximum lengths, and this 
>model is unlikely to change in the forseeable future, excessive traffic 
>bursts will be more likely subject to drop-tail policing than other 
>kinds of more smoothly shaped traffic. More than that, the bursty 
>traffic will not only have less chance of reaching its target with all 
>fragments in place, but it will also most probably do much harm to

Note that these are not IP fragments. Dropping one of them from a burst
is no different from dropping one from any other random flow.

Also don't forget the end2end path covers two endpoints on the host side
so we must consider host side requirement too. This looks like a tug of
war between the host side favoring large burst vs the network side
favoring small burst. Perhaps TOE can be a good bridge between the two.

Rgds,

Jerry

Also 
>other innocent and otherwise well-behaving flows as well.
>
>Marko
>
>
>>
>> Jerry
>>
>> Sr. Staff Engineer
>> Solaris Networking & Security Technology
>> Sun Microsystems, Inc.
>>
>> >I don't have a good answer but going much higher than 16Kbytes MTUs
>> >seems unlikely... and at 10Gig this is still close to 100kpps.
>> >
>> >	cheers
>> >	luigi
>

---

From: Marko Zec <[email-masked]>
Date: Sat, 6 Mar 2004 11:21:37 +0100

On Saturday 06 March 2004 17:48, H.K. Jerry Chu wrote:
> >> much buffering will be needed. Routers that do cut-through need
> >> not buffer the whole pkt. More significant is the drop of the DRAM
> >> price and increase of the DRAM size making memory a non-issue in
> >> many cases today.
> >
> >implementing unlimited packet queues, we would already have seen
> > tons of such devices on the market over the past 10 years or so.
> > Excessive queuing delays on routers are bad, especially for
> > high-speed traffic managed by TCP-like congestion control schemes.
> > The queue lengths on routers have always been and will always
> > present a tradeoff between a smallest usable window for handling
> > traffic bursts, and the desire of keeping the queueing delays as
> > low as possible on the other hand.
>
> Assuming pkt switching speed has scaled up too, I don't see why
> today's much faster routers with a larger max queue length will cause
> the average queuing delay to be any longer than routers 10 years ago.

Congestions and associated queuing delays typically arise due to finite 
line speeds at outgoing network interfaces, which are sometimes offered 
more traffic than they can transmit, so the remaining packets have to 
be either queued or discarded. This phenomenon is well known and is 
largely independent of the switching fabric performance.

So, when such congestions do occur, the max. queuing delay will be 
approximately equal to the max. buffer length (in bits) divided by the 
line card speed. Note that with today's faster line speeds we cannot 
afford to have the similar level of queuing delays as on routers 10 
years ago - those delays have now to be reduced (preferably) 
proportionally to the increase in line rates. And that can't be 
enforced with unlimited buffer lengths.

> traffic using 9K jumbo frames vs a non-conforming, traffic burst of
> 13 back-to-back regular Ethernet frames. They both require the same
> amount of memory bandwidth in the forwarding path (2*9K ~= 13*1500).
> If all the forwarding optimization in today's routers has made the
> switching speed of 13 consecutive Ethernet "cells" from the same flow
> similar to 2 back-to-back jumbo frames, why would the former be any
> more evil than the latter?

> Note that these are not IP fragments. Dropping one of them from a
> burst is no different from dropping one from any other random flow.

It _is_ different, because with spiky bursts, you won't be occasionaly 
losing only a packet or two, but you can expect to loose the entire (or 
most of the) burst at once.

Marko

---

From: Craig Partridge <[email-masked]>
Date: Mon, 08 Mar 2004 11:15:05 -0500

>buffer slots available for queuing further packets / bursts.

I must be missing something here.  I think Alexandre's generally right.
If the buffers load and empty according to the amount of data in them,
then packet size isn't an issue.

The only situations in which this isn't true, that I can think off the
top of my head, are:

    * If the router has places where it moves all the bytes in the packet
      buffer, regardless of how much data the buffer actually contains.
      Since this would cause extraordinary internal bandwidth challenges,
      I doubt anyone does it for any but the lowest speed routers.

    * If the router has a deep pipeline and the time at each stage in the
      pipeline is set to equal the average packet time rather than the
      minimum packet time.  But most folks carefully design router pipelines
      for minimum packet times precisely because of this risk.

So clue me in -- what detail of router design am I missing?

Craig