Subject: RE: Reordering and MPLS?

From: Shahram Davari <[email-masked]>
Date: Tue, 4 Jan 2000 09:00:55 -0800

Hi,

No. Preserving the order of packets is not an MPLS requirement. In fact in
some cases such as E-LSP setup (explained in draft-ietf-mpls-diff-ext-02, in
which one LSP can carry multiple COS traffics), packets will be reordered
based on their COS.

Shahram Davari
Systems Engineer, Product Research
PMC-Sierra Inc. 
555 Legget drive,
Suit 834, Tower B,
Ottawa, ON K2K 2X3, Canada
Phone: +1 (613) 271-4018
Fax  : +1 (613) 271-7007

> -----Original Message-----
> From: [email-masked] [mailto:[email-masked]]
> Sent: Tuesday, January 04, 2000 10:35 AM
> To: [email-masked]
> Subject: Reordering and MPLS? 
> 
> 
> 
> Is there a requirement to preserve packet order in MPLS?  
> That is, are MPLS
> enabled network elements (individual routers or a collection 
> of routers in
> a single location) required to forward IP packet in the order they are
> received?
> 
> Thanks
> ======================================================================
> Dr. Victor S. Frost                    Sabbatical contact information:
> 				       Sprint
> Dan F. Servey Distinguished Professor  Phone: 913 534 5334
> Electrical Engineering                 Fax:     913 534 2526
> and Computer Science	               [email-masked] 
> Information and Telecommunication     9300 Metcalf  
>   Technology Center		       Overland Park, Kansas 66212    	
> University of Kansas			      	
> 2291 Irving Hill Dr.
> Lawrence, Kansas 66044-7541
> Phone: (785) 864-4833  FAX:(785) 864-7789
> e-mail: [email-masked]   http://www.ittc.ukans.edu/
>

---

From: Bora Akyol <[email-masked]>
Date: Tue, 4 Jan 2000 20:31:22 -0800

Reordering of IP packets inside MPLS packets will clearly cause most TCP
applications to attempt early transmits for sessions thereby reducing
goodput of the network.

At least in our box, we are making every effort possible such that we don't
reorder packets MPLS or IP belonging to the same flow. I believe that as
long as TCP is not changed to handle this, all MPLS/IP router vendors have a
responsibility to guarantee ordered arrival packets belonging to the same
flow in order to preserve the Internet.

Diffserv has gone to a lot of trouble assuring this for AF classes (as in
dictating how queueing should be done). 

I expect that all MPLS equipment out there will do their best to keep
packets in order.

Yes I know it is a pain to do this at higher speeds but oh well. I would
suggest that we add this to the MPLS architecture document.

Bora Akyol
Pluris (http://www.pluris.com)

-----Original Message-----
From: Jeremy Lawrence [mailto:[email-masked]]
Sent: Tuesday, January 04, 2000 7:45 PM
To: [email-masked]
Subject: Re: Reordering and MPLS?

Reordering has come up a couple of times recently, and I'm
wondering why. As Noel has eloquently explained, a
well-designed network attempts wherever feasible to deliver
similar packets (e.g. packets of the source, destination, and
diffserv class) in order, and a well-designed end-system can
tolerate occasional mis-ordering. Unless I'm missing something,
this all seems quite sensible.

Is it just that a note to this effect needs to be added to a
document somewhere? (I'm not sure where: these are fundamentals
of packet network & equipment design independent of MPLS or IP.)

Or is it that people are considering applications of MPLS in
circumstances where it seems desirable that packet ordering be
guaranteed at the network layer? If so, then it seems important
to discuss the applications themselves.

Jeremy Lawrence

---

From: Bora Akyol <[email-masked]>
Date: Wed, 5 Jan 2000 14:08:36 -0800

No I meant the first. In order delivery should be **guaranteed**.
Otherwise, out of order delivery causes problems at the TCP level and
reduces
your network goodput. This is especially true with fast retransmit.

Bora Akyol

-----Original Message-----
From: J. Noel Chiappa [mailto:[email-masked]]
Sent: Wednesday, January 05, 2000 10:13 AM
To: [email-masked]
Cc: [email-masked]
Subject: RE: Reordering and MPLS?

    > From: Bora Akyol <[email-masked]>

    > all MPLS/IP router vendors have a responsibility to guarantee ordered
    > arrival packets belonging to the same flow

There's a big difference between "guarantee to never do <X>" and "do <X>
very, very rarely". From this next comment I gather you meant the second,
though?

    > I expect that all MPLS equipment out there will do their best to keep

This is entirely reasonable.

    > I would suggest that we add this to the MPLS architecture document.

Sure, as long as it's "do their best", not "guarantee".

Of course, if you leave out a specific numberic target, then you run the
risk
of some smart-aleck saying "oh, our box does reorder one packet in ten, but
we are doing our best". However, as pointed out on the IETF list in another
thread, the IETF doesn't get into saying whether implementations meet a spec
anyway, so even if the spec did say "don't reorder more than .001% of
packets", it wouldn't do that much good. (And the argument about what number
to put in could and possibly would go on forever! :-)

	Noel

---

From: "J. Noel Chiappa" <[email-masked]>
Date: Wed, 5 Jan 2000 21:02:46 -0500

>> There's a big difference between "guarantee to never do <X>" and "do
    >> <X> very, very rarely".

    > reduces your network goodput. This is especially true with fast
    > retransmit.

Well, you do concede that an endnode has to be able to handle segments
arriving out of order, right? (This is because it's basically impossible to
tell whether you're seeing a retransmitted packet, or a delayed packet: e.g.
host A sends packets 1, 2 and 3; packet 2 gets lost; host A retransmits 2; so
host B sees packets 1, 3, 2 arrive.)

So, given that, it's purely a "maximizing throughput" argument you're making,
right?

However, a reordering will not cause more of a hit to throughput than a lost
packet, right? (Extensive analysis of end-end behaviour in both circumstances
to determine the exact performance impact omitted, since the weak ordering
above will do for my argument. Also omitted is speculation about how to
changes to fast retransmit to filter out "out-of-order" incidents.)

So, if the rate at which the network reorders packets is much smaller than
the rate at which it drops packets (which I expect to be true), how does
completely getting rid of all reordering buy you a noticeable increase in
throughput? And why expend all the effort to absolutely, utterly guarantee
that there is no reordering - when it doesn't buy you anything noticeable?

Again, I have no problem with a mandate that MPLS boxes try hard to never
reorder packets. But I find a *guarantee* of absolutely no reordering i) of
great expense (in terms of complexity, mechanism, etc), and ii) of no real
worth.

	Noel

---

From: Bora Akyol <[email-masked]>
Date: Wed, 5 Jan 2000 18:54:28 -0800

OK,

I see your point. I would reword it to say that packet ordering should be
preserved under normal operating conditions. When packets are getting
rerouted in the network or even within a box due to a failure, then we will
see reordering.

I also agree that reordering such that fast retransmit within TCP is not
triggered, we should be OK.

So if this translates to "very rarely" than I agree. On the other hand, I
would like to say that striping packets through a switch without any respect
to packet order is NOT acceptable as long as applications are not changed to
handle this.

Thanks

Bora

-----Original Message-----
From: J. Noel Chiappa [mailto:[email-masked]]
Sent: Wednesday, January 05, 2000 6:03 PM
To: [email-masked]
Cc: [email-masked]
Subject: RE: Reordering and MPLS?

host A sends packets 1, 2 and 3; packet 2 gets lost; host A retransmits 2;
so

So, given that, it's purely a "maximizing throughput" argument you're
making,
right?

packet, right? (Extensive analysis of end-end behaviour in both
circumstances

	Noel

---

From: "J. Noel Chiappa" <[email-masked]>
Date: Sat, 8 Jan 2000 23:47:22 -0500

> From: Christian Kuhtz <[email-masked]>

    >> Switches should not make a practise of reordering packets (except as
    >> explicitly called for by resource allocation mechanisms, and in those
    >> cases it will be between flows).

    > And a flow is what exactly? ... "between flows" to me implies some
    > amount of automagic detection of flows.

I wasn't intending to mean anything specific by "flow", just "a stream of
packets that the resource allocation treats as a group". So for "flows" in the
above sentence, substitute "packets in different resource allocation
groupings", or whatever similar phrase will keep you happy.

	Noel