Subject: RE: [Simple] MSRP Boundary header

From: "Orit Levin" <[email-masked]>
Date: Thu, 1 Jul 2004 18:13:06 -0700

The argument below about copying in parallel with checking is true for
single-purpose devices whose only or main job is to processes MSRP
messages.  In general-purpose OS, copying occurs in lower layers (e.g.
network drivers) and by the time the data reaches the application, it is
already out of the cache.

After some investigation, below are the facts:
 
The described behavior has been noted by OS vendors and some of them
already use special version of copy instruction that does not pollute
the cache when copying network data (e.g. using movnti on X86).

Furthermore, there are protocols under development (CISCO along with
Microsoft are by the way among the founders of RDMA consortium -
www.rdmaconsortium.org) that facilitate DMA directly from the wire into
the application data buffers (so-named Remote DMA).  This obviously
completely bypasses the cache and makes the bottleneck argument
obsolete.

Finally, if one is implementing a gateway between MSRP and other
protocols, it may need to copy the whole data only to insert additional
escape sequences on the output path.

To summarize this, I would say that it will be strange if SIMPLE comes
up with a brand new protocol which is "optimized" for
- Dedicated HW devices
- Encourages lazy implementation where the data is not being chunked
into reasonably sized blocks so that the relays in the middle have hard
time to decide whether they will be able to process a single chunk or
not
- and still being called the protocol for Instant Messaging 

Orit.

> -----Original Message-----
> From: [email-masked] 
> [mailto:[email-masked]] On Behalf Of Cullen Jennings
> Sent: Thursday, July 01, 2004 11:05 AM
> To: Chris Boulton; Ben Campbell; Vijay Kishen Hampapur Parthasarathy
> Cc: Ted Hardie; Paul H Kyzivat; [email-masked]; [email-masked]
> Subject: Re: [Simple] MSRP Boundary header
> 
> 
> This whole discussion is premised on the idea that framing by 
> finding a length header is somehow much more efficient, 
> faster, etc than the boundary framing. I like to poke at that 
> a little. This efficiency is probably only relevant on relays 
> or things dealing with lots of messages and these run on 
> machines where the CPU has a cache.
> 
> The experiments I have been playing with show that given the 
> boundary recommendations that we are making that allow you to 
> check for a "----"
>  4 bytes at a time allow you look for boundary at the same 
> rate that the memory can be copied. Basically the bandwidth 
> from main memory to cache is the bottleneck and the extra 
> check if the 4 byte value is "----" takes no extra time - the 
> CPU is mostly idle waiting for the cache to fill. If you send 
> a message which consisted of a special constructed message 
> that looked very similar to the boundary, this might slow 
> down but this is an artificial corner case I can't get worked 
> up about.
> 
> I know of no reason for MSRP to prefer the length to the 
> boundary scheme. If someone has one, let's get that on the 
> table and establish if there is a real need to do both. The 
> downside about both is interoperability issues when people 
> only implement one.
> 
> I point out that I can find boundaries at the same rate I can 
> copy memory and this is way way way beyond the total network 
> bandwidth that can flow in or out of my machine. I'm missing 
> what the argument for doing both is.
> 
> Cullen
> 
> 
> 
> On 7/1/04 12:27 AM, "Chris Boulton" <[email-masked]> wrote:
> 
> > I agree with Ben - why do something twice.  This only increases 
> > complexity and potential to provide conflicting + syntactically 
> > incorrect semantics.
> > 
> > Chris.
> > 
> > 
> > -----Original Message-----
> > From: Ben Campbell [mailto:[email-masked]]
> > Sent: 30 June 2004 19:35
> > To: Vijay Kishen Hampapur Parthasarathy
> > Cc: Ted Hardie; Paul Kyzivat; [email-masked]; [email-masked]
> > Subject: Re: [Simple] MSRP Boundary header
> > 
> > Because having two ways to do the same thing adds 
> complexity to both 
> > the
> > 
> > implementation and the specification.
> > 
> > Vijay Kishen Hampapur Parthasarathy wrote:
> > 
> >> Can you comment on the initial question on why not allow 
> both methods 
> >> for message framing.
> >> 
> >> Vijay
> >> 
> >> -----Original Message-----
> >> From: Ben Campbell [mailto:[email-masked]]
> >> Sent: Wednesday, June 30, 2004 6:20 AM
> >> To: Paul Kyzivat
> >> Cc: Ted Hardie; [email-masked]; Vijay Kishen Hampapur 
> >> Parthasarathy; [email-masked]
> >> Subject: Re: [Simple] MSRP Boundary header
> >> 
> >> 
> >> 
> >> Paul Kyzivat wrote:
> >> 
> >> 
> >>> 
> >>> Ted Hardie wrote:
> >>> 
> >>> 
> >>>> At 7:48 PM -0700 6/29/04, Sean Olson wrote:
> >>>> 
> >>>> 
> >>>>> Why not allow both methods?
> >>>>> It seems clear that there are probably two common use cases:
> >>>>> 
> >>>>> 1) Simple, brief messages as in today's IM conversations
> >>>>> 2) Larger, more complex, richer messages as forseen in 3G and 
> >>>>> other environments
> >>>> 
> >>>> 
> >>>> 
> >>>> Sorry to ask such a potentially silly question, but aren't the 
> >>>> messages in case one to be handled by SIP/SIMPLE, rather 
> than MSRP?
> >>> 
> >>> 
> >>> No. Maybe if you have a one-shot message it should be. But if you 
> >>> are having a conversation composed of simple brief 
> messages then I 
> >>> believe
> >> 
> >> 
> >>> the expectation is that you would be using MSRP.
> >>> 
> >> 
> >> 
> >> Or at one composed of a series of short, interactive 
> messages. (Which
> > is
> >> probably what Paul meant to say.
> >> 
> >> 
> >>> At least that is my expectation. One subject that has yet to be 
> >>> dealt with is the relationship (and possibly migration) 
> between page 
> >>> mode and session mode. The fact that you ask the question 
> you did is 
> >>> an indication that this isn't a solved problem.
> >>> 
> >> 
> >> 
> >> This is true. We have had discussions indicating that we 
> need to say 
> >> something about when to use each, and how to transition 
> between them.
> >> 
> >> I personally think that sort of thing belong in the SIMPLE
> > architecture
> >> draft effort, which we have not been putting much effort 
> into of late.
> >> 
> >> 
> >>>    Paul
> > 
> > _______________________________________________
> > Simple mailing list
> > [email-masked]
> > https://www1.ietf.org/mailman/listinfo/simple
> > 
> > 
> > This message has been scanned for viruses by MailControl - 
> > www.mailcontrol.com
> > 
> > _______________________________________________
> > Simple mailing list
> > [email-masked]
> > https://www1.ietf.org/mailman/listinfo/simple
> > 
> 
> 
> _______________________________________________
> Simple mailing list
> [email-masked]
> https://www1.ietf.org/mailman/listinfo/simple
> 

_______________________________________________
Simple mailing list
[email-masked]
https://www1.ietf.org/mailman/listinfo/simple

---

From: Cullen Jennings <[email-masked]>
Date: Thu, 01 Jul 2004 22:44:26 -0700

This is a very strange argument, I think you exactly support my point but
then you seem to draw the opposite conclusion. I will respond to you points
inline but this whole argument does not make sense to me. I can scan for
boundary at the speed of the memory bandwidth on any any machine anyone is
likely to run a relay on. This is many giga bits per second worth of data. I
can not imagine that this will represent a significant percentage of time
for any relay and removing it will not make a relevant difference.

More inline

> the cache when copying network data (e.g. using movnti on X86).

Yep - any my point was you could transfer it from block A in memory to block
B in exactly the same time that you can transfer it and check it. If I used
the non cache polluting instructions to transfer the data into a register,
check it, then transfer out, I can still do this at the full bandwidth of
the memory bus. 

> completely bypasses the cache and makes the bottleneck argument

Yes very familiar with this. Let's see what would happen,  a fragment of UDP
packet arrives on the Ethernet card and you are going to know where to DMA
the right portion of this fragment into a buffer in user space that
represents the correct portion of a TCP data stream. I don't think so. If
this protocol ran over UDP, you might be able to use this to improve
performance beyond the current vector IO stuff that NT supports. However, it
does not run over UDP and the DMA trick is not going to help for this.

Even if it did, your performance would be total limited by the bandwidth of
the NIC cards - lets say you somehow had really a lot of of really fast NIC
cards, you are going to hit the limit of the PCI bus to DMA it to memory.
This time to scan it is going to be a drop in the bucket because the CPU
memory bandwidth is so much better than the memory to NIC bandwidth which is
probably much greater than the actual bandwidth the NIC cards can put onto
the network.  
 
> escape sequences on the output path.

Uh, perhaps, if you implement TCP and MSRP in VLSI on the NIC card but
otherwise I think the machine is going to copy it to memory.

> up with a brand new protocol which is "optimized" for

I'm lost how it is "optimized" for this

> time to decide whether they will be able to process a single chunk or

Again - don't see how you get to this conclusion.

> - and still being called the protocol for Instant Messaging

Let's say the type of IM you are interested in results in an average MSRP
message that is 256 bytes long. If we take a 3Ghz P4 with 800 MHz frontside
and good DDR memory, how many million of these messages do you think I can
boundary scan per second?

How many million messages per second do you think you can relay on a single
host? 

Do you think this result in dropped performance? I don't think it will drop
performance at all because you can't relay from one stream to another stream
with zero copies but even if you did implement it as another pass over the
data just to do this, I doubt it can make any real difference.