Subject: Issues with draft-radoaca-ppvpn-gvpls-01.txt

From: Ali Sajassi <[email-masked]>
Date: Sun, 13 Apr 2003 20:09:29 -0700

One one the main issues with GVPLS is the use of different PWs for unknown 
unicast frames versus known unicast frames (e.g., using PtMP PWs for 
unknown unicast and MPtP PWs for known unicast).  I raised my concern 
during the WG meeting and I will reiterate it here again.

In regular LAN network, in absence of failure, there is NO possibility for 
packet re-ordering. However, with the use of different PWs for unicast 
frames in GVPLS, the possibility of re-ordering customer's Ethernet frames 
exists even in absence of any failures (e.g., during steady state network 
operation). This out-of-sequencing happens when a customer destination MAC 
address is not learnt and customer's frames gets flooded via PtMP PW. Now 
if after transmission of the first frame, the PE learns the destination MAC 
address, then the subsequent frames go over a different PW (e.g., MPtP). 
Since two consecutive frames can go over different PWs and these different 
PWs can take different path w/ different delays in the network, the two 
frames can arrive out-of-order at the destination PE and some protocols can 
be susceptible to this out-of-ordering of the packets.

I'd like to know how this packet out-of-ordering will be addressed in GVPLS.

-Ali

---

From: "himanshu shah" <[email-masked]>
Date: Mon, 14 Apr 2003 07:39:27 -0400

response in line.

-----Original Message-----
From: Ali Sajassi [mailto:[email-masked]]
Sent: Sunday, April 13, 2003 11:09 PM
To: [email-masked]
Cc: Ali Sajassi
Subject: Issues with draft-radoaca-ppvpn-gvpls-01.txt

operation).

HS> This may not be entirely true. One can expect, unknown/broadcast frame
HS> to arrive out-of-order even in LAN Network. This happens due to how
HS> frame processing is architected in a given box. The flooding frame
HS> (i.e. broadcast/unknown) within the box may be treated differently
HS> than a known frame going out on a single port (e.g. separate queue).

This out-of-sequencing happens when a customer destination MAC
address is not learnt and customer's frames gets flooded via PtMP PW. Now
if after transmission of the first frame, the PE learns the destination MAC
address, then the subsequent frames go over a different PW (e.g., MPtP).

HS> This is possible but rare. Note that in order to have this happen
HS> that remote unknown station has simultaneously generated a frame which
HS> causes the local PE to learn its address AND there is a second frame
HS> generated by the source without having to get a response back on the
HS> frame sent earlier AND this second frame gets through the network over
HS> the other PW just enough faster to beat to the first frame AND this
HS> second frame is important enough to have it arrive in order. The point
HS> is that this is rare enough to be insignificant.

HS> If one is using LDP for outer tunneling (most common case), both
pseudowires
HS> maps to same path for a given two PE endpoints. Actually, there is
HS> a greater and more common risk of out-of-sequence of packets
HS> in a single pseudowire due to network topology changes, fast reroute,
etc.
HS> Also, most protocols are capable of handling packets received
out-of-order,
HS> especially, when that occurs during session initiation time.

regards,
himanshu

-Ali

---

From: "amita" <[email-masked]>
Date: Mon, 14 Apr 2003 09:01:19 -0700

Response inline.

regards,
-Amita
----- Original Message -----
From: "Ali Sajassi" <[email-masked]>
To: <[email-masked]>
Cc: "Ali Sajassi" <[email-masked]>
Sent: Sunday, April 13, 2003 8:09 PM
Subject: Issues with draft-radoaca-ppvpn-gvpls-01.txt

> packet re-ordering.
***********************************
Not entirely true.There is a possibility (may be rare) of frame re-ordering
in L2 networks , when the switch architecture supports separate unicast and
multicast queues.
The queue arbitration for multicast and unicast queues could cause the
similar situation that you are describing below, when the slower unicast
frame can arrive after the destination MAC is learnt.
802.1D does not mandate frame ordering between the unicast and multicast
destination MAC.
***********************************
> However, with the use of different PWs for unicast
> address is not learnt and customer's frames gets flooded via PtMP PW.

>Now
**************************************************
This could very much happen .
As far as final delivery of frames is concerened at the End Point equipment
, it is a responsibility of the protocol operation to ensure that single
path is used between the source and destination station pair.( as per
802.1D ).
It is possible to ensure that by always using the same tunnel LSP between
PEs for unicast and multicast PWs within a VPN.
**************************************************

---

From: Ali Sajassi <[email-masked]>
Date: Mon, 14 Apr 2003 08:55:26 -0700

>HS> than a known frame going out on a single port (e.g. separate queue).

That would exasperate the problem even more. So if some of the existing 
platforms have different internal processing path that would result in 
packet re-ordering between unknown unicast versus known unicast, then we 
should fix the problem instead of amplifying it by allowing it to be done 
everywhere in the network (both edge and core).  Also I would like to know 
where in IEEE standards, it allows for such behavior. We can take this 
discussion to IEEE list.

>HS> is that this is rare enough to be insignificant.

With the simultaneous transmission by the end-points, this situation is not 
as rare as you may think !! Are you implying that with this type of VPLS 
service, we should allow different SLA based on the transmission.

>HS> maps to same path for a given two PE endpoints.

Even with this, you can still have out-of-ordering because of ECMP !!

>Actually, there is

As I said before, this out-of-ordering happens in steady state for GVPLS 
which is different from existing LAN operations. Given this is a LAN 
emulation service, lets see if this is acceptable by IEEE folks.

-Ali

---

From: "himanshu shah" <[email-masked]>
Date: Mon, 14 Apr 2003 12:42:48 -0400

responses in line........

-----Original Message-----
From: Ali Sajassi [mailto:[email-masked]]
Sent: Monday, April 14, 2003 11:55 AM
To: [email-masked]; 'Ali Sajassi'; [email-masked]
Subject: RE: Issues with draft-radoaca-ppvpn-gvpls-01.txt

HS> Out-of-order packets between unknown-unicast/multicast and
HS> known unicast is well known and well accepted fact. Perhaps
HS> you should inquire this on IEEE mailing list.

HS> Perhaps you would like to give an example of a particular
protocol/scenario
HS> that you are thinking of where such simultaneous transmission
HS> at session initiation is so problematic for a rare possibility
HS> of out-of-order?

HS> Thank you. You give one more example of how an out-of-order
HS> delievery could occur even when traffic is going over single
HS> pseudowire.

HS> The point is out-of-order can happen in GVPLS in very rare
HS> circumstances.  If you look at RSTP, out-of-order is mentioned
HS> as a possibility during topo changes as an acceptable risk against
HS> greater benefits. I suppose one can apply same logic here.

regards,
Himanshu

-Ali

---

From: Ali Sajassi <[email-masked]>
Date: Mon, 14 Apr 2003 10:16:17 -0700

> > In regular LAN network, in absence of failure, there is NO possibility for

We are talking about frame ordering for unicast destination MAC (known 
versus unknown).

However, you cannot ensure a single path even if you use the same tunnel 
LSP because of load balancing for ECMP as I indicated previously.

-Ali

---

From: Ali Sajassi <[email-masked]>
Date: Mon, 14 Apr 2003 10:44:12 -0700

>HS> you should inquire this on IEEE mailing list.

Not for the steady state. If you look at the IEEE 802.1D standards it says:

"NOTE 1 —The Forwarding Process in Bridges (7.7)does not mis-order or 
duplicate frames.
Where Bridges in a Bridged LAN are capable of connecting the individual 
MACs in such a way that
multiple paths between any source station –destination station pairs 
exist,the operation of a protocol is required to ensure that a single path 
is used.
NOTE 2 —Frame misordering and duplication (6.3.4) does not occur during 
normal operation."

It clearly states that during normal operation misordering DOES NOT OCCUR !!

>HS> delievery could occur even when traffic is going over single

No !!! traffic doesn't get re-order when a single PW is used in case of 
ECMP. It can get re-order when there are multiple PWs are involved for the 
same flow  (as in the case of GVPLS).

>HS> greater benefits. I suppose one can apply same logic here.

As mentioned before, IEEE 802.1D standards allows for possibility of 
reordering in case of topology changes but not in steady state operation.

Regards,
Ali

---

From: "himanshu shah" <[email-masked]>
Date: Mon, 14 Apr 2003 14:41:51 -0400

response in line.
  -----Original Message-----
  From: Ali Sajassi [mailto:[email-masked]]
  Sent: Monday, April 14, 2003 1:44 PM
  To: [email-masked]; 'Ali Sajassi'; [email-masked]
  Subject: RE: Issues with draft-radoaca-ppvpn-gvpls-01.txt

    That would exasperate the problem even more. So if some of the existing
    platforms have different internal processing path that would result in
    packet re-ordering between unknown unicast versus known unicast, then we
    should fix the problem instead of amplifying it by allowing it to be
done
    everywhere in the network (both edge and core).  Also I would like to
know
    where in IEEE standards, it allows for such behavior. We can take this
    discussion to IEEE list.

  Not for the steady state. If you look at the IEEE 802.1D standards it
says:

  "NOTE 1 -The Forwarding Process in Bridges (7.7)does not mis-order or
duplicate frames.
  Where Bridges in a Bridged LAN are capable of connecting the individual
MACs in such a way that
  multiple paths between any source station -destination station pairs
exist,the operation of a protocol is required to ensure that a single path
is used.
  NOTE 2 -Frame misordering and duplication (6.3.4) does not occur during
normal operation."

  It clearly states that during normal operation misordering DOES NOT OCCUR
!!

  [himanshu shah] Despite what it says, today in normal bridges, broadcast
(unknown unicast frame is a flood and hence it is a broadcast) and unicast
frames between two stations may get out of order for the reasons I described
before; irrespective of the state of the network. Again, I invite you to
site an example where a miniscule possibility of such an out-of-order spells
disaster for a protocol (at session initiation time).

  No !!! traffic doesn't get re-order when a single PW is used in case of
ECMP. It can get re-order when there are multiple PWs are involved for the
same flow  (as in the case of GVPLS).
  [himanshu shah] If one was to do ECMP using the bottom most label, then it
is true that out-of-order would not occur for a single PW. However, is it
mandated that ECMP must be done on bottom-most label (i.e. PW label which
technically could be below several labels)? Lets for a moment accept that
such out-of-order would occur for 2 PWs. I would like to see an example
where a rare case of mis-order between broadcast-path frame and unicast-path
frame.

  As mentioned before, IEEE 802.1D standards allows for possibility of
reordering in case of topology changes but not in steady state operation.
  [himanshu shah] topo changes in small bridged network as compared to topo
changes in big-I network are at quite different scale. If one was to compare
the number of out-of-order packets due to topo-changes in big-I network to
out-of-order packets due to 2 PW, one would wonder where the real problem
is.

  Regards,
  Ali

---

From: "amita" <[email-masked]>
Date: Mon, 14 Apr 2003 21:08:33 -0700

Just to close on the clarification for this email thread ... ( .. response
inline )

regards,
-Amita

----- Original Message -----
From: "Ali Sajassi" <[email-masked]>
To: "amita" <[email-masked]>; <[email-masked]>; "Ali Sajassi"
<[email-masked]>
Sent: Monday, April 14, 2003 10:16 AM
Subject: Re: Issues with draft-radoaca-ppvpn-gvpls-01.txt

> We are talking about frame ordering for unicast destination MAC (known
[Amita ] Sure , I meant unknown unicast MAC that gets translated to  VLAN
flooding , the traffic that goes on multicast queues in some architectures.

> > > if after transmission of the first frame, the PE learns the
destination
> >MAC
> > > frames can arrive out-of-order at the destination PE and some
protocols
> >can

---

From: Ali Sajassi <[email-masked]>
Date: Tue, 15 Apr 2003 08:10:25 -0700

>frames between two stations may get out of order for the reasons I 
>described before; irrespective of the state of the network. Again, I 
>invite you to site an example where a miniscule possibility of such an 
>out-of-order spells disaster for a protocol (at session initiation time).

So, you're acknowledging that it is in violation of IEEE standards but 
you're saying that it is not a big deal because some of the existing 
switches currently do that and so it is O.K. to use different paths for 
(unknown and known unicast) within the network. There are holes in this 
argument:
a) bridges come in all different price range and if some low-cost bridges 
don't adhere to IEEE standards that's a different thing
b) what is important is the differential delay between the two paths. So, 
inside a bridge, the differential delay can be in order of micro-seconds or 
milliseconds; whereas, in the network the differential delay path can be 
much larger (function of network and path congestion). And the larger the 
delay path, the bigger the issue.
c) now the re-ordering is a function of not only network availability 
(failure) but also a function of network congestion (which is more frequent)

Also although IP protocols are not susceptible to packet re-ordering, there 
are legacy protocols (e.g., NetBEUI) that are susceptible to this 
re-ordering. We can take this discussion to IEEE to find out what is the 
tolerable differential delay in such cases.

>where a rare case of mis-order between broadcast-path frame and 
>unicast-path frame.

There are currently deployed routers that perform load balancing based on PWs.

>changes in big-I network are at quite different scale. If one was to 
>compare the number of out-of-order packets due to topo-changes in big-I 
>network to out-of-order packets due to 2 PW, one would wonder where the 
>real problem is.

Topology changes in SP core as well as SP access networks are function of 
link/node failure which is tied up with end-to-end availability numbers; 
however, intermittent path/node congestion in steady state is lot more 
frequent than the link/node failure.

-Ali