Subject: [tcpm] Are you interested in TOEs and related issues (Resend)

From: Sunay Tripathi <[email-masked]>
Date: Tue, 2 Mar 2004 00:09:00 -0800 (PST)

Hi,

AT the end of the TSVWG session on Monday, I was trying to see if people
are interested in discussing TOE (TCP offload Engines) from the
prespective of new technology and issues arising out of it. I am
not sure if there are any protocol level issues yet that need to be
discussed under the scope of IETF (tcpm) charter. I just want to
scope out the interest in this area from implementation and integration
in the host OS point of view.

After the meeting, I already met 5-6 people who are interested in
talking about it so we will try and get together sometime wed/thur
after the plenary. If there are more people interested, Ted can
get us a room or figure out a more structured discussion either this
or next IETF meeting.

Please drop me a note with you time preference if you are interested 
and I'll see what we can rig up. A brief background and possible list 
of things to discuss is listed below.

TOE BAckground
--------------

In last couple of years, the number of vendors selling TOE based hardware
has increased significantly and we at SUN have personally talked to 
about 2 dozen such vendors. The claim is that TOE will be necessary for
10Gb NICs and for emerging technologies like iSCSI, RDMA, etc. These
vendors already have first generation h/w available for Windows and Linux
while some of them have moved on to 2nd generation. On Linux, these 
vendors support TOE by overwriting the S/W stack with their own hooks.
With pretty impressive performance and reduction in CPU utilization claims,
even the end users have started asking (atleast SUN) about TOE support.

Some things to discuss
----------------------

1) Are you working with TOEs in any way?
2) Have you faced any issues with TOE. Someof them are:
        - failover. When TOE runs out of resources or doesn't understand
          some options/features.
        - Old (and possibly buggy implementations) burned in ASIC which
          will be hard to upgrade as new modifications are made to TCP
          (even if they are 2-3 every year).
        - End to End correctness issues like TOE acking a packet but dying
          before the data is transfered to host memory. Or accepting the
          data to be sent out but dying before the packet hits the wire.
3) Any other issues?

Cheers,
Sunay

-- 
Sunay Tripathi
Senior Staff Engineer,
Solaris Kernel Networking,
Sun MicroSystems Inc.

email: [email-masked]                 Phone: 650-786-6007 (W)

_______________________________________________
tcpm mailing list
[email-masked]
https://www1.ietf.org/mailman/listinfo/tcpm

---

From: Craig Partridge <[email-masked]>
Date: Tue, 02 Mar 2004 10:07:35 -0500

Well, the interesting thing is we went down this path in the early 1980s
and found that TOEs didn't work.

If I try to distill all that we learned in the 1980s into one question, I
come out with:

    In the 1980s we discovered that communicating with a TOE over a bus
    about a TCP connection was as expensive or more expensive than
    simply handling the TCP connection in the main processor.  What about
    today's TOE designs makes them different?

Craig

---

From: Rick Jones <[email-masked]>
Date: Mon, 08 Mar 2004 10:26:55 -0800

> TOE has become a buzz-word that needs to be qualified.  Some folks
> include Checksum Offload and Large Segmentation Offload when talking
> about TOE.  When others talk about TOE, they refer to offloading the
> whole TCP/IP stack.

Humbly submitted:

Pinky TOE => Checksum Offload (CKO)
Middle TOE => CKO + LSO (aka TSO or large send)
Big TOE => Full TCP Offload

rick jones
-- 
Wisdom Teeth are impacted, people are affected by the effects of events.
these opinions are mine, all mine; HP might not want them anyway... :)
feel free to post, OR email to raj in cup.hp.com  but NOT BOTH...

---

From: Barney Wolff <[email-masked]>
Date: Mon, 8 Mar 2004 14:52:06 -0500

> Big TOE => Full TCP Offload

You forgot these:

Stubbed TOE => disabling TOE so tcpdump/snoop works
Broken TOE  => bug in firmware, difficult to diagnose

-- 
Barney Wolff         http://www.databus.com/bwresume.pdf
I'm available by contract or FT, in the NYC metro area or via the 'Net.

---

From: Sunay Tripathi <[email-masked]>
Date: Tue, 9 Mar 2004 13:43:31 -0800 (PST)

[Sorry I was out travelling and in the meanwhile this thread has
morphed into router design :) Anyway, I hope someone can start a
new thread with the router design subject while we continue on
TOE discussion - Thanks]

David,

> I wouldn't say in general that we found out that TOEs didn't work.  I 
> think a more qualified comment of what we found out was that in an 
> environment where the network is the bottle neck, TOEs did not provide 
> any cost effective benefits; and as CPUs got faster, the TOEs just 
> provided costs without benefits. :-)
> 

I think we are talking about offloading the whole stack or to the point
where the NIC needs to keep the state. The stateless offload (cksum
offload and LSO has already been happening for a while).

> With 10GigE, we are in the realm where the network is not the bottle 
> neck.  Even with GigE, the CPU has to do significant work to keep the 
> card fed, leaving little time for doing any application work.  I think 
> there are very few if any CPUs available today that can keep a 10GigE 
> pipe fed, much less handle an inbound 10GigE stream.

Our experience as well. The fastest intel or sparc CPU can saturate the
1GigE but there is not much room left for doing meaningful work by the
application.

> Features such as Checksum Offload and LSO are features that have low 
> impact on the OS and its TCP/IP stack.  Other features such as having a 
> timer to delay and aggregate transmit interrupts also help to reduce 
> the amount of work that the CPU has to do to process the data.
> 
> For me, the question is what can be done in a 10GigE NIC to allow the 
> host CPU to handle both sending and receiving full speed on that 
> interface?  TOE is just one part of that discussion and the danger is 
> that if too much focus is put on TOE, the other aspects of how to 
> improve how the CPU and the NIC communicate will not be properly 
> investigated.

You defintely have to fanout to distribute load across multiple CPUs if
you don't have TOE and CPU speed doesn't scale dramatically in couple of years.
We are developing a dynamic polling/interrupt mechanism to use general
purpose CPU to saturate 10GigE. Using TOE based cards is another possible
approach (you still need to deal with moving a giga byte of data across
your back plane though).

Cheers,
Sunay

> 
> 			-David Borman
> 
> On Mar 3, 2004, at 12:07 AM, Craig Partridge wrote:
> 
> >

---

From: "David P. Reed" <[email-masked]>
Date: Tue, 09 Mar 2004 18:29:39 -0500

As the network gets faster, the user perceived performance issue moves into 
the application layer, except for the special case of remoting bulk disk 
drive accesses.   There is a small "opportunity window" of app requirements 
where the TCP layer is the crucial issue for performance.

It's easy to get focused on unrealistic benchmark cases that ignore the 
complexity of app logic, and the power of general purpose computing 
architectures that can dynamically partition workload rather than 
statically partitioning function into boxes.

I wouldn't waste my time on TOEs.  There are more useful things to work on, 
like making end-to-end secure and resilient communications practical.

---

From: Sunay Tripathi <[email-masked]>
Date: Wed, 10 Mar 2004 18:23:30 -0800 (PST)

David,

Thanks for clarifying. 

> At 06:29 PM 3/9/2004, David P. Reed wrote:
> >I wouldn't waste my time on TOEs.  There are more useful things to work 
> >on, like making end-to-end secure and resilient communications practical.
> 
> To clarify this, if it seems like a non-sequitur or change in topic, 
> remember that offloading TCP into a separate box locks in any security 
> flaws into the architectural structure you had to create to make it 
> happen.  In practice, this means that no one will have the energy or money 
> to fix the security and resiliency issues in such solutions - which means 
> more kludges in wrap-around devices and interceptors that create the false 
> illusion of security.

Yes, that was one of the first thoughts that came to my mind when I
started looking at this space. Its not just security issues but bugs and
new features. TCP despite being mature still sees couple of RFC or tweaks
a year that needs to be rolled out to existing systems. Apparantly the
TOE vendors had thought about that. The firmware running the protocol code
actually lives on the host and is delivered via exisiting delivery
mechanism (packages in Solaris which can be upgraded). When machine boots
and interface is configured, the protocol firmware is sucked in. So
delivering fix or new revs in pretty similar to exisiting mechanisms.
Also, the source code is still 'C' based so people like me can still
work with it (just need to use some magic in the end to create the
firmware instead of complier). Some vendors we talked to have promised
that they can pretty much suck in the same code that runs on the host
into the card (as long as we can clean some complier specific directives
etc). Although I must admit that this aspect (common source files) of TOE 
has not been investigated by us at all so far.

Cheers,
Sunay

> 
> Communications networking is a systems problem, and ignoring the systems 
> issues by focusing on a narrow problem space is not a good investment of 
> effort.
> 
>

---

From: Rick Jones <[email-masked]>
Date: Thu, 11 Mar 2004 10:49:10 -0800

> has not been investigated by us at all so far.

That seems like a path out when the TOE is based on an embedded CPU, but doesn't
seem like it would work for those TOEs that are ASICs.

And a mostly rhetorical question - strictly in the context of TCP comms and not
RDMA and the like - if one can code tightly enough for an embedded CPU a TCP/IP
stack including comms with the host across the I/O bus with enough performance
to be reasonable for a given link, can't one presumeably make the host's TCP/IP
stack similarly efficient?  If nothing else that seems to be alluded to by the
last sentence in the quoted paragraph.

rick jones

---

From: "David P. Reed" <[email-masked]>
Date: Wed, 10 Mar 2004 09:55:18 -0500

---

From: David Borman <[email-masked]>
Date: Thu, 4 Mar 2004 16:12:37 +0900

Craig,

			-David Borman

---

From: Sunay Tripathi <[email-masked]>
Date: Thu, 11 Mar 2004 11:31:59 -0800 (PST)

Rick,

> seem like it would work for those TOEs that are ASICs.

The TOEs I am talking about have 1 or more embedded ARM processor (seems to 
be the more common model these days). 

> last sentence in the quoted paragraph.

I think Craig did ask this question before and the benefit is not so much
from the protocol processing saving but from the data movement. Below is
a portion of my earlier reply to Craig's email.

"
1) Extra processor(s) buried in the TOE for networking processing which is
   hidden from the kernel and leaves the host CPU to do more application
   related work. Saves the cost of licences for application which take
   number of CPU into account (oracle is one such application cited).
2) On low end (1-2 CPU) x86 based machines, cost of adding a processor
   is much higher than adding a TOE (I personally haven't verified this).
3) For the up and coming 10Gb NICs, TOE will help saturate the link. Some
   vendors assert that TOE will be required to support 10Gb NICs.
4) Performance reasons. Just the LSO aspect of TOE (sending large chunks of
   data and letting the TOE split it up in mss size pieces) and ack
   coalescing gives a pretty good boost (our own prototypes indicates that
   this is true). The gains are by optimizing data movement and not by
   offloading protocol processing.
5) TOE is necessary for RDMA, iSCSI etc. for layering reasons. I am not
   involved with RDMA so someone who is an expert can probably comment on
   this part.
6) TOE based NIC are already making pretty good headway in embedded space.
   The technology is already maturing so why not use it in broader market.
"

Cheers,
Sunay

---

From: Rick Jones <[email-masked]>
Date: Mon, 08 Mar 2004 12:16:56 -0800

> Broken TOE  => bug in firmware, difficult to diagnose

Yes, well... if we are going for full humour (actually, I wasn't just trying to
be funny but still... :)

TOE jam => the kruft of a TOE solution
Hammer TOE => one where the offload is to an embedded CPU
TOE nail => when offloaded security/hardening is included
Athlete's Foot => when a TOE becomes security compromised
Tip TOE => when it operates in a stealth mode
Foot => what you have when more than one TOE are bonded together.
TOE shoes => the shims interfacing a TOE with the rest of the system/stack.

I suppose that had development of the Digital 2114X chips continued, one might
have spoken of a Tip TOE through the Tulips.

ad nauseum :)

rick jones

---

From: Craig Partridge <[email-masked]>
Date: Sun, 07 Mar 2004 12:43:26 -0500

In message <[email-masked]>, David Borman writes

>provided costs without benefits. :-)

I take a stronger view, but also agree we need to clarify.

>about TOE.  When others talk about TOE, they refer to offloading the 

I understood TOE as being offloading just about the entire stack.

I would agree that experience shows that certain functions can be offfloaded
very effectively, including checksums and some memory management.

I also agree with your thought question -- namely which functions to offload
in a high performance environment -- is the right one.

Craig