Subject: Comments on the L2 VPN framework and solutions documents

From: Muneyoshi Suzuki <[email-masked]>
Date: Wed, 16 Apr 2003 10:41:10 +0900 (JST)

I have a couple of comments on Emulated LAN which consists of VPLS 
forwarders and full mesh pseudo wires with split-horizon forwarding 
scheme, proposed in the L2 VPN framework as well as related solutions 
documents, such as draft-lasserre-vkompella-ppvpn-vpls-04.txt and 
draft-kompella-ppvpn-vpls-01.txt.

(1) Protection racing

This scheme does not terminate STP, so if a pseudo wire in the mesh 
fails, it may be restored with a MPLS protection mechanism. In this case, 
there is a possibility that the recovery time of the protection mechanism 
may affect failure detection of user-STP.

For example, if an LSP in the mesh fails, LSP protection mechanism may 
recover it, however, if user-STP detects the failure simultaneously, 
then it re-organize the L2 topology in a few minutes; but after the 
re-organization, it also detects recovery of the failed LSP, then 
user-STP restore the original L2 topology in a few minutes.

Finally, this wastes twice time of STP topology change and spoils 
fast MPLS protection mechanism. It is too unrealistic and dangerous to 
prohibit user-STP, it may leads broadcast loop, thus, I think interworking 
between user-STP and MPLS protection mechanism is needed.

Note that, the above discussion assumes that the user-STP is working 
properly in the above situation. However, this assumption is doubtful.
See the next comment.

(2) Black hole

From a viewpoint user-STP, Emulated LAN, which consists of VPLS 
forwarders and full mesh pseudo wires, is logically equivalent with a 
hub-repeater. Assume that the single SP network consists of PE Bridges 
and a single hub-repeater, and a PE Bridge port is connected to a port 
in the hub-repeater.

In this case, if the hub-repeater entirely fails (that is, *all* LSPs 
fail at the same time), user-STP detects it and may re-organize user L2 
topology, if possible (e.g., the user has a back door link). If a single 
port in the hub-repeater fails (that is, *all* LSPs connected to a single
PE fail at the same time), user-STP also detects it and may re-organize 
the topology.

However, failure of a single LSP in the mesh is logically equivalent 
with the situation that the communications between two ports in the hub-
repeater fail, *but the remains are normal*. I'm not sure what happens 
in this case, because I think this kind of failure is not assumed in STP 
protocol design. I do not rigidly quantify what happens, but think there 
will be a black hole.

MPLS protection/FRR mechanism does not solve this issue, because it can 
minimize probability of partial failure, but it does not eliminate the 
possibility. If the LSP recovery is unsuccessful, the user will encounter 
fatal situation.

I believe the VPLS solution must be robust, that is, it must recover or 
lead the network to a safe status automatically however the network 
encounters a rare but fatal error. To enable large scale deployment, it must 
not force the operators and users to reboot boxes even if the worst case.

Thanks,

Muneyoshi Suzuki
Nippon Telegraph and Telephone Corp.

---

From: Norman Finn <[email-masked]>
Date: Thu, 17 Apr 2003 16:53:22 -0700

Muneyoshi,

The problems of running STP or RSTP over a "soft" connection such as
an emulated LAN is a problem that has been well explored, for example
in the context of ATM LAN emulation.  The common critical item among
these failure scenarios is that the users of the "soft" connection
never have reliable knowledge of whether the connection is up or down.

IEEE 802.1AD will certainly address these issues.  In particular,
802.1AD is assuming, at this time, that the customers' spanning trees
will be carried by a network of Provider Bridges which is, itself,
running either Rapid Spanning Tree or Multiple Spanning Trees.  In
general, if one is running spanning tree over spanning tree in this
manner, or if one is running spanning tree over any other connection
that fails and heals itself over a period of a few seconds or longer,
one runs into the kinds of problems you mention.

If the Provider's network is composed solely of point-to-point
physical links running RSTP or MSTP, and if certain other constraints
such as the number of Provider's BPDUs allowed per second are chosen
with this problem in mind, then one may expect the customers' spanning
trees to work properly.

In general, if the underlying soft connection technology heals itself
in less than a second or so, there should be no significant problems
with the customers' spanning trees.  If connectivity can be fully or
partially broken for more than a few seconds, then the customer must
slow down his spanning tree failure detection timers correspondingly.
There is a gray area between one second and a few seconds.

One of the goals of the L2-VPN effort must be, in my opinion, to
specify how full meshes can be guaranteed to meet the sub-second
restoration requirement, or at the very least, to define what guarantees
meshes can meet.  Alternatively, a mechanism must be provided to signal
failures of the full mesh in such a way that all of the connected
Provider Bridges know that the link has failed, and avoid it until it
can be re-established.

In any event, but especially if a failure signaling mechanism is
the best that IETF can do, then IEEE 802.1AD will have to consider
very carefully whether and/or how a similar mechanism can be used to
ensure that the customer, in turn, reliably detects any interruption
or restoration of the service.

At this time, interworking between the Provider and Customer networks
has not received a lot of attention in 802.1, but it has not been
ruled out, except for the obvious conclusion that they must not simply
be peers.

-- Norm

---

From: Muneyoshi Suzuki <[email-masked]>
Date: Fri, 18 Apr 2003 16:27:12 +0900 (JST)

Norm,

Thanks for reply.

> running either Rapid Spanning Tree or Multiple Spanning Trees.  

Good news.

> In general, if one is running spanning tree over spanning tree in this

Yes. That is one of my points.

> with this problem in mind, then one may expect the customers' spanning

Agreed. If SP does *not* use full-mesh split-horizon scheme, it is true.

> There is a gray area between one second and a few seconds.

In general situation, it is true.

> meshes can meet. 

My point is the network behavior when the link protection is unsuccessful.
If the SP network consists of Bridges and point-to-point links running
STP, there is no significant issues. However, if the SP use full-mesh 
split-horizon scheme, customers may encounter loop or black hole.

> Alternatively, a mechanism must be provided to signal

I'm not sure the above sachem. Do you mean if one of path in the mesh
fails, the Provider Bridges blocks *all* the paths connected to the 
Bridge until the path recovery? I agree this is one of solutions, but
don't think it is realistic.

Thanks,

Muneyoshi Suzuki

---

From: Norman Finn <[email-masked]>
Date: Fri, 18 Apr 2003 16:46:24 -0700

Muneyoshi,

Thank you for bringing up the subject.  It needs more attention.
This (long) message is an attempt to give it some.  :)

The requirement that a full mesh must meet is that you never have
partial connectivity for more than a brief period of time.  Otherwise,
the mesh is not suitable to server as part of a network that carries
bridged traffic, including the various 802.1D Layer 2 control protocols
such as spanning tree BPDUs.

I will note, as an aside, that in general, a full mesh may be
carrying Customers' BPDUs, a Provider's BPDUs, or both.  The
requirements are the same, in any case.

Following the requirements terminology, let's call the full mesh an
emulated LAN, and the module inside a box that passes the data between
the pseudowires and the emulated port the Forwarding Function.  The
bridge sits above the emulated port.

One additional concept is required.  The emulated port which the FF
provides the upper layers has certain characteristics, among
them an ifOperState, which is either UP or DOWN.  One would assume
that, upon initialization, the FF would be ifOperState == DOWN to
the upper layers until it had obtained (through discovery) its list
of other FFs and had established its pseudowires.  At that point, it
would change its state to ifOperState == UP, which would trigger the
bridge above it to start normal operations.  This would typically
involve transmitting BPDUs and waiting to see if any other bridges
were also transmitting BPDUs, before it starts to send and receive
data.  This is all part of the standard way that bridges prevent
loops; 802.1D specifies exactly what a bridge must do when a link's
ifOperState goes UP or DOWN.

Now, the rules:

 1. Every Forwarding Function has a list of the other FFs it thinks
    belong to its emulated LAN.  This list is obtained through the
    discovery process.

 2. Similarly, every Forwarding Function has a list of the FFs with
    which it has established functional pseudowires.  This may involve
    control-layer (not Ethernet data layer!) pings.

 3. The Forwarding Function, of course, signals, accepts, and/or
    deletes pseudowires as needed to make the two lists match.

 4. Unless or until an FF has a pseudowire up and running to every
    member of its list, it must present ifOperState == DOWN to its
    upper layers.  That is, any time that an FF's list of established
    pseudowires does not match its list of emulated LAN participants,
    it must stay ifOperState == DOWN.  When one or both of the lists
    has changed so that they match, again, the FF may present
    ifOperState == UP.  (There may be other constraints on ifOperState
    as well, for example, the ifAdminState control.)

Of course, data may flow through an emulated LAN port in either
direction only when ifOperState == UP.  The above rules mean, in
in essence, that an FF doesn't pass data unless it thinks that it
has a full mesh established.  Furthermore, it signals the bridge
via ifOperState whenever the state of the full mesh changes.  This
guarantees that the bridge running over the full mesh will work
properly.

It is particularly important to note that this "do not send or receive
unless all links are up" rule is *not* imposed on the individual
pseudowires, but on the emulated LAN port!  The emulated ifOperState
must not interfere with establishing a pseudowire and verifying that
it works.  Also, one FF does *not* know whether another FF to which it
has established a pseudowire is or is not passing data up or down its
stack.  Without this decoupling, one gets into a horrible tangle of
intention vs. reality and race conditions.

With this set of rules, an emulated LAN matches the semantics of a
fat yellow coax, or of a set of hubbed Ethernet segments.  In fact,
it works better than a physical medium!  This set of rules fixes a
problem of shared media: connecting a new device may cause a temporary
loop, and consequent broadcast storm.  Since all FFs do a down-then-
up transition of ifOperState when a new FF joins or an old one leaves,
the unsignaled joins and leaves which cause temporary loops cannot
occur.

In my earlier e-mail, I claimed that a full mesh must be able to
guarantee that it will either heal itself within a specified period
of time, or must notify the upper layers (bridges, in this case) of
the failure.  With the formulation above, we may restate this
constraint in a manner that one may believe can be met:

  Whenever the set M of FFs belonging to a full mesh changes, all
  FFs belonging to M after the change must be reliably notified of
  the change within time T.

(Where "reliably" means "often enough that I can put up with the
consequences when it fails".)

I'm sure I can optimize this further, so that most connectivity
transitions may go unsignaled in most FFs.  This would improve the
connectivity of the network without compromising its integrity.  In
particular, when an FF dies or leaves, there is no requirement to
immediately tell everyone, or to make them transition ifOperState.
But the essential point is that I'm pretty sure that it is possible
to make the full mesh compatible with bridging.

The task for 802.1AD will be to ensure that a similar guarantee
can be offered to the Customer whose spanning tree is running over
a network of Provider Bridges, given that the full mesh at the core
of the Provider Bridge network can notify the Provider Bridges of
its state.

-- Norm

---

From: Muneyoshi Suzuki <[email-masked]>
Date: Mon, 21 Apr 2003 13:48:00 +0900 (JST)

Norm,

> them an ifOperState, which is either UP or DOWN. 

This deeply depends on physical link characteristic such as full-duplex,
half-duplex bi-directional, and uni-directional. Especially, in an 
uni-directional link such as LSP, it is difficult to timely notify a failure
from receiver side to sender side, because reverse path may not available.

> has a full mesh established.

I understand your proposal is if an LSP between two PE fails, block
all LSPs connecting to these PEs.

Is it wise to use links as serial system?

Here, we assume N as number of PEs, and R as reliability of a single LSP.
In this case, reliability of a single PE in the proposal is:

R ^ {2*(N -1)}.

Obviously, it is ruinous under large N.

I think there is a much simple and robust solution. Connect Provider Bridges
with point-to-point LSPs and use STP/RSTP in the network, and don't use 
split-horizon scheme. Needless to say LSPs must be raw mode; tagged mode
violates the standard. 

In this case, if PEs are connected with full mesh LSPs, reliability of a 
single PE is:

1- {(1- R^2) ^ (N -1)}.

Obviously, this is a robust solution because it use LSPs as parallel system.
Furthermore, this does not need to revise the existing standards, it only
needs Q-in-Q or MAC-in-MAC specification in 802.1ad.

Thanks,

Muneyoshi Suzuki

---

From: Ali Sajassi <[email-masked]>
Date: Mon, 21 Apr 2003 16:13:50 -0700

Muneyoshi,

>split-horizon scheme. Needless to say LSPs must be raw mode; tagged mode

There are several issues w/ this appraoch:

1)  Scalability aspect of the service - e.g., having tens or hundreds of 
thousands of customers which translate into 10Ks/100Ks PE-VLANs. It should 
be noted that 802.1ad in based on 802.1D/Q with 12-bit vlan field. Also it 
is not as simple as increasing the VLAN field, since it impacts the 
802.1s/w and GMRP algorithms.

2) Impact to existing IEEE algorithms such as MSTP, RSTP, GMRP, GVRP, etc 
if need more than 4K vlan space.

3) Creating interdependency of different access domain - e.g., the same 
PE-VLAN needs to be used in different access domains

4) Inter-operability with MPLS access networks - how do you interoperate 
between a QinQ access network and an MPLS access network that doesn't run STP

5) Mandating the use of STP over MPLS/IP core - although some service 
providers don't mind running STP within an access network (e.g., QinQ 
island), they are concern running STP between islands because of its 
network mgmt aspect of it.

-Ali

---

From: Eric Rosen <[email-masked]>
Date: Tue, 22 Apr 2003 10:42:47 -0400

Norm> Unless or until an FF has  a pseudowire up and running to every member
Norm> of its [auto-discovered] list, it  must present ifOperState == DOWN to
Norm> its upper layers.

Then a  single flaky PE  would bring down  the entire emulated LAN,  and the
emulated LAN  would stay  down until either  that PE  comes back up,  or the
discovery  procedure  successfully  removes   that  PE  from  the  list  and
successfully informs all the other PEs.  This doesn't seem that good.

---

From: Eric Rosen <[email-masked]>
Date: Tue, 22 Apr 2003 10:39:29 -0400

> may affect failure detection of user-STP. 

I think the only feasible "solution" for this would be for the user-STP time
constants to  be appropriately  adjusted to the  recovery time frame  of the
PSN.  It helps to use a PSN which has a shorter recovery time frame, but all
we can really do is note the relationship.  

> However, failure of a single LSP  in the mesh is logically equivalent with
> the  situation that  the  communications  between two  ports  in the  hub-
> repeater  fail, *but  the remains  are normal*  ... I  think this  kind of
> failure is not assumed in STP protocol design. 

Recovery from  this should be  at the  PSN layer, I  think, not at  the VPLS
layer. 

> I  think there  is a  much simple  and robust  solution.  Connect Provider
> Bridges  with point-to-point  LSPs and  use STP/RSTP  in the  network, and
> don't use split-horizon scheme.

You're proposing  that all the VPLS  traffic should be confined  to a single
spanning tree,  (with the resulting sub-optimal routing),  to compensate for
the case in which  the PSN's routing can't get traffic from  A to B, but can
get traffic from  A to C and from C  to B?  That's a very  large cost for an
extraordinarily small benefit.

---

From: "Cheng-Yin Lee" <[email-masked]>
Date: Tue, 22 Apr 2003 11:38:42 -0400

Eric,
That's the point, a single flaky PE (in full-meshed with split-horizon)
would cause improper LAN emulation, bringing down customer/access
network bridging/routing (even if we don't do what Norm Finn is
proposing here).
Hopefully you would acknowledge the dilemma here.

Thanks,
Cheng-Yin

---

From: Eric Rosen <[email-masked]>
Date: Tue, 22 Apr 2003 13:57:43 -0400

Cheng-Yin> a single flaky PE (in full-meshed with split-horizon) would cause
Cheng-Yin> improper  LAN emulation,  bringing  down customer/access  network
Cheng-Yin> bridging/routing (even if we don't do what Norm Finn is proposing
Cheng-Yin> here).

I  don't think  that's  been shown.   If  a particular  PE  is not  behaving
properly, then there  are going to be issues  communicating with the systems
that are behind it.  I don't understand why this brings the customer's whole
network down. 

Cheng-Yin> Hopefully you would acknowledge the dilemma here. 

I  agree that  there are  unresolved problems  regarding the  effect  of PSN
events on  the customer's bridged network.   I don't know  whether these are
solvable or not, just that more work needs to be done. 

(Of  course, all  these problems  can  be eliminated  by making  all the  CE
devices be routers, and using an IPLS service instead ;-))

---

From: "himanshu shah" <[email-masked]>
Date: Tue, 22 Apr 2003 15:05:55 -0400

I agree with Eric on two points,

1) Verifying FF synchronization using low-grade, high-level
   loopbacks (such as ping), is difficult aspects of
   VPLS

2) Using IPLS instead and relying on CE router
   mechanisms to handle individual loss of connectivity

/himanshu

---

From: Ali Sajassi <[email-masked]>
Date: Tue, 22 Apr 2003 17:44:46 -0700

>successfully informs all the other PEs.  This doesn't seem that good.

It also seems to me that taking the entire Emulated LAN (set of PWs) down 
and up as the result of a PW or node failure, might be too much (e.g., 
causing unnecessary service interruption). With respect to failures 
affecting the Emulated LAN, there can be two types:

a) PW failure: PW failure should be taken care by PSN - if not, then there 
is something wrong with the node and it should be consider as node failure, 
right ?
b) node failure: If there is a redundant PE, then redundancy mechanism 
takes care of it (e.g., intra-island STP) and if PE is not redundant, then 
it gets removed from the mesh but the remaining PEs still maintain the 
full-mesh with each other.

It seems if the customer BPDU timer is adjusted to compensate for these two 
caeses (PW recovery by PSN and node recovery by SP STP), then it should be O.K.

-Ali

---

From: Muneyoshi Suzuki <[email-masked]>
Date: Wed, 23 Apr 2003 11:53:29 +0900 (JST)

> Recovery from  this should be  at the  PSN layer, I  think, not at  the VPLS

Neither PNS nor VPLS layer guarantee recovery. So, these are not solution.

> extraordinarily small benefit. 

It is much better than the fragility solution that has loop, blackhole or 
unreliable flaky PEs problems.

Thanks,

Muneyoshi Suzuki

---

From: Eric Rosen <[email-masked]>
Date: Wed, 23 Apr 2003 09:36:30 -0400

Eric> Recovery from  this should be  at the PSN  layer, I think, not  at the
Eric> VPLS layer. 

Muneyoshi> Neither PNS nor VPLS layer  guarantee recovery. So, these are not
Muneyoshi> solution.  

There is  no way to guarantee  recovery against every  possible failure.  If
you're sending the A-->C traffic on the path A-->B-->C, becuase that path is
on the spanning tree, you might still have a problem in B where it drops the
traffic that arrives from A and is destined to C.  I'd say that this failure
scenario is at  least as likely as the other,  probably more likely, because
it involves more VPLS mechanisms. 

Eric> You're proposing  that all  the VPLS traffic  should be confined  to a
Eric> single  spanning tree,  (with the  resulting sub-optimal  routing), to
Eric> compensate for the  case in which the PSN's  routing can't get traffic
Eric> from A to B, but can get traffic  from A to C and from C to B?  That's
Eric> a very large cost for an extraordinarily small benefit.

Muneyoshi> It  is much  better than  the fragility  solution that  has loop,
Muneyoshi> blackhole or unreliable flaky PEs problems. 

That's not much of a cost-benefit analysis.

---

From: Muneyoshi Suzuki <[email-masked]>
Date: Thu, 24 Apr 2003 11:39:03 +0900 (JST)

> That's not much of a cost-benefit analysis.  

Good point. The most expensive is network maintenance labor costs.
The worst case is the network is down due to corner-cutting implementation,
then someone is awaken by midnight phone, and run around to reboot boxes.

Thanks,

Muneyoshi Suzuki

---

From: Norman Finn <[email-masked]>
Date: Fri, 25 Apr 2003 09:02:18 -0700

Muneyoshi,

> from receiver side to sender side, because reverse path may not available.

I don't think we're referring to the same model.  The correct model
for how a bridge interacts with full-meshes, one full-mesh per VPLS
instance, is:

            |===========================================--------+
            |                   |          | forwarding |       |
            |                   |          |  function  +---    |
   E  i   --+                   | untagged +  VPLS 100  |       |
   t  n     |      Provider     |          | for BPDUs  +---    |  P  r
   h  t     |       Bridge      |          |------------|       |  s  o
   e  e     |                   |          |            +---    |  e  u
   r  r   --+                   |          | forwarding |       |  u  t
   n  f     |     Each "+" in   |  VLAN 26 +  function  +---    |  d  i
   e  a     |    the border of  |          |  VPLS 200  |       |  o  n
   t  c     |     this block    |   mux-   |            +---    |  w  g
      e   --+   represents one  +  demux   |------------|       |  i
      s     |     bridge port   | function |            +---    |  r  f
            |                   |          |            |       |  e  u
            |                   |          |            +---    |  s  n
          --+                   |          | forwarding |       |     c
            |                   | VLAN 589 +  function  +---    |  i  t
            |                   |          |  VPLS 300  |       |  n  i
            |                   |          |            +---    |     o
          --+                   |          |            |       |     n
            |===========================================--------+
                                A          B            C

The single interface above the "A" label at the bottom is the Provider
Bridge's bridge port that opens to the L3 world.  The three interfaces
above the "B" label are the interfaces between the forwarding functions
and the mux-demux unit that translates between the Provider Bridge's
P-VLAN space and the VPLS VCID space.  The "C" ports are the mouths of
the pseudowires.  The physical ports on the routed side are of no
interest to the bridge.  :)

In my last note, I was pointing out that the each forwarding function
MUST control its "B" interface so that it provides 1) full service, or
2) no service.  (Let's be honest -- my last note was a little confused
between the "A" interface and the "B" interfaces.)

Since VPLS meshes have one characteristic that physical LANs don't --
one VLAN can be disconnected, while another is working -- IEEE 802.1
needs to examine this situation, and figure out how to deal with it.

Finally, let me note that only one VPLS -- the one used for BPDUs --
is absolutely required to prevent a meltdown of the Provider's network.
A failure of one of the other VPLSs affects only that customer's traffic.

> Obviously, it is ruinous under large N.

We differ, if at all, only in our definition of "large".

> needs Q-in-Q or MAC-in-MAC specification in 802.1ad.

As Ali points out, the problem with this approach is that the Provider Bridge
must either send or receive one BPDU on each pseudowire.  If a Provider Bridge
is serving 2000 VPLSs, each with 10 pseudowires, that's 20,000 BPDUs per
second.  That would be a difficult task to accomplish.  On the occasion when
a failure occurs, and there are 20,000 state changes to process, I worry that
the pseudowire spanning tree would melt down, as did networks with slow
processors some years ago.

-- Norm

---

From: Norman Finn <[email-masked]>
Date: Fri, 25 Apr 2003 09:47:43 -0700

Eric,

I tend to agree.  After more thought (I mention this in an earlier long message),
I decided that the only full mesh that must follow this rather drastic paradigm
is one that carries BPDUs.  If BPDUs don't get everywhere they're supposed to
get, then whatever network is depending on those BPDUs, Provider or Customer,
will either loop or partition, and looping is more common than partitioning in
this kind of failure.  Fortunately, most VPLSs won't be carrying BPDUs, or if
they are, they won't matter, since the customer is not multi-homed.  But, for
multi-homed customers' BPDUs, or for the Provider's BPDUs, it's all or nothing.

-- Norm

---

From: Norman Finn <[email-masked]>
Date: Fri, 25 Apr 2003 09:48:43 -0700

I second the motion.  :)

---

From: Norman Finn <[email-masked]>
Date: Fri, 25 Apr 2003 09:52:08 -0700

Ali,

No matter what, no matter how good our failure detection/recovery mechanisms
are, there will always be short periods when connectivity is not correct.
If we can bound those periods (and that was the point of my first long message),
then cranking up the spanning tree timers will solve everything.  If we cannot
bound those periods (e.g. the full mesh can never be completed, because of some
erroneous ACL in some router somewhere) then nothing will work.

-- Norm

---

From: Yakov Rekhter <[email-masked]>
Date: Fri, 25 Apr 2003 09:56:16 -0700

Norm,

> they are, they won't matter, since the customer is not multi-homed.  

How do you know that for most VPLSs "the customer is not multi-homed" ?

Yakov.

---

From: Norman Finn <[email-masked]>
Date: Fri, 25 Apr 2003 15:24:52 -0700

Yakov,

You don't know, for sure.  On the other hand, there are some reasonable
monetary incentives for a Provider to classify the Customers into those
which are multi-homed, and those which are not.  The reason is that
uni-homed Customers require no BPDU snooping and no forgetting MAC addresses
due to events in the Customers' networks.  If I'm wrong, and most customers
are multi-homed, and are running BPDUs, then each such customer's full-mesh
must be operated in all-or-nothing mode.

-- Norm

---

From: Ali Sajassi <[email-masked]>
Date: Mon, 28 Apr 2003 19:42:13 -0700

>erroneous ACL in some router somewhere) then nothing will work.

Yes I agree, but this is independent from which VPLS model is being used. 
In other words, the same problems that can exist with VLAN emulation model 
(using the full mesh of PWs) can also exist with the virtual port model 
where a partial mesh of PWs is used with STP over them (e.g., running STP 
over the core network between different islands). So I don't see any 
advantages of running STP over the core in this regard but the 
disadvantages in terms of scalability as mentioned previously are clear.

-Ali

---

From: Muneyoshi Suzuki <[email-masked]>
Date: Wed, 30 Apr 2003 17:13:12 +0900 (JST)

Norm,

> for how a bridge interacts with full-meshes, one full-mesh per VPLS

The discussion is in the context of WG last call of the L2 FW doc and
how to progress related solution drafts. So, I'm referring Eric's model
describe in the FW doc. The following model completely differ from Eric's,
so the following is disagreement to the FW doc, isn't it?

(1) It seems to me, in the above model, PEs are interconnected with per 
customer meshes, where the PEs attached to the customer are interconnected 
with a full mesh. Therefore, in the worst case, there 2000 full meshes 
between PEs, if number of customers are 2000. Originally, full mesh has the
O(N^2) problem, so the proposed scheme has O(#ofCustomers * N^2) problem. 
Furthermore, if fast protection is supported to avoid the protection 
racing problem and partial mesh, there are double meshes between the PEs. 
So, I don't feel any reality from the above model.

(2) Frankly, I don't well understand the above model, because it 
illustrates some implementation, but is not a protocol architecture.
In the Bridge architecture defined in a series of the IEEE standards,
a Bridge consists of MAC entities, a MAC relay entity, and a Higher
layer entity. Could you clarify where these entities are located in 
the above figure? Where are MSAPs defined in ISO/IEC 15802-1 is located? 
If the provider Bridge is a VLAN aware Bridge, where are the E-ISSs located?
What is VLAN mux-demux function? According to the standards, the VLAN ID 
value is effective only inside the MAC relay entity, so it does not 
identify MSAP. I'm not sure whether this conforms with the standards or not,
but I think to realize this architecture big changes are needed to the 
current standards.
------

> If you use spanning tree among the pseudowires connecting Islands,
> as in:
>                        pw A
>      island 1 --------------------- island 2
>            \                         /
>        pw B \_______       _________/ pw C
>                     \     /
>                     island 3
> then one of pw A, pw B, or pw C must be blocked.  Let's suppose it's
> pw C.  Then, all traffic from island 2 to island 3 must be relayed
> by island 1.

> In other words, since wires are almost "free" in this space, one
> tends to have a lot of them.  It the extreme case that you have a
> full mesh of connections among all the participants in a net, spanning
> tree is a particularly wasteful approach; it guarantees to block all
> but N of the N*(N-1)/2 pseudowires!  (: Yes, a bridge-bigot is saying
> that spanning tree is not optimal for all interconnect plans. :)

If xSTP (STP/RSTP/MSTP) is running in the SP network, there is no need to 
interconnect PEs with full mesh. But it has the link block problem; some 
links are blocked and unused in the network. However, as I mentioned in 
the mailing list, the full mesh scheme proposed in the FW doc has a 
similar problem, if it supports fast protection to avoid the protection 
racing problem and partial mesh. It needs double full meshes and 50% are 
not usually used. This problem is not much different from xSTP link block 
problem.

And if the per customer meshes are used and if fast protection is supported,
it wastes too many paths. I think xSTP link block problem is much 
better than it.

------

> If you configure a relatively sparse interconnect plan, using just
> a few pseudowires, and if those pseudowires approximate physical
> reality, then spanning tree becomes more viable.  The scaling
> problem for 3000 VPLS instances at 10 pseudowires each becomes 3000
> VPLS instances at 2-3 pseudowires each, reducing the BPDU load from
> 20,000 to 7,500.  I would remark that our best bridges top out at
> somewhere on the order of 1,000 - 2,000 BPDUs per second, at this
> time.  I need to think a bit more before I agree that this solution
> "simple and robust".

I'm not sure what is the issue in the above contexts. If you are claiming
that the full mesh scheme/per customer meshes don't need to process customer
BPDUs and xSTP based SP network need process all of customer BPDUs, 
thus the former is scale, it is completely nonsensical.

In my understanding, in the SP network, customer BPDUs are

o forwarded transparently and the network does not look up BPDUs contents.
o forwarded transparently but the network snoop BPDUs and update states.
o terminated and discarded regardless BPDUs contents.
o terminated and discarded but the network snoop BPDUs and updates states. 
o terminated and discarded but a new BPDU is created based on the BPDU.

And the SP network may create a BPDU and insert for a customer.

Anyway, in the following, how to process xSTP is discussed because
most interests relate on it.

(1) The SP must ensure a single tree topology in the SP network. Manual
configuration, provider-STP, and provider-RSTP can support it without
any difficulties. In this process, the SP don't care customer-xSTP, 
because it is the SP's responsibility to ensure a tree topology, thus it 
is determined under the SP's policy and control. It is a quite unrealistic 
scenario that the SP network topology is changed by the customer-xSTP.

(2) It is customer responsibility to ensure a single tree topology in the
customer network which consists of customer sites interconnected by the
SP network and backdoor links. This is because, the customer can cause 
a loop whether the SP forwards customer-xSTP transparently or the SP
provides per-customer xSTP instances. If the customer does no use
xSTP, there is a possibility to cause a loop.

(3) If the SP forwards customer-xSTP transparently, and if the customer 
use customer-xSTP, it can avoid a loop even if it pass through the SP 
network. This is because, the SP ensures a single tree topology and a 
single customer Bridge that supports customer-xSTP in a loop is sufficient 
to detect and cut the loop. 

(4) However, it is highly desired to detect a customer loop in the SP 
network. But, (a) detection is not limited xSTP, another technology
can also detect a loop, (b) provider core Bridges don't need to support 
per-customer xSTP instances, because Provider Edge Bridges that support 
pre-customer xSTP instances are sufficient to detect and cut the loop, 
and (c) the full-mesh scheme also has completely the same issues.

(5) And it is highly desired for the SP network to detect customer network 
topology change and clear the cache for the customer, if the customer use
xSTP and is multihomed. In this case, provider Bridges need to snoop 
customer-xSTP then forwards it transparently. However, both the xSTP based 
SP network and full-mesh scheme have completely the same issues.

In summary, I don't find any disadvantage in the xSTP based SP network
scheme compare with the full-mesh scheme. Furthermore, it does not have
black hole, loop, and flaky PEs problems, because it based on the standards.

Thanks,

Muneyoshi Suzuki