Subject: Re: [forces] BasicMetadataDispatch WAS(Re: LFB LPM issue

From: "Chuanhuang Li" <[email-masked]>
Date: Wed, 13 Apr 2011 18:47:14 +0800

BasicMetadataDispatch is defined as a general metadata dispatch LFB initially.
It can be used wherever user wants to dispath packets according some special metadata.
It has more flexibility.
 
Yours,
Chuanhuang 
	
======= 2011-04-13 18:23:18 Jamal Hadi Salim, wrote: =======

>Following up after looking at the draft some more..
>In Figure1 is the sole purpose of BasicMetadataDispatch to just select
>a mediaport path? It doesnt seem sensible given we could use
>LFB-port-groups to select LFB (instances).
>
>cheers,
>jamal
>
>On Wed, Apr 13, 2011 at 5:32 AM, Jamal Hadi Salim <[email-masked]> wrote:
>
>> That was my initial thinking - but now that you bring it up again,
>> you made me look closer at figure1...  Other than EtherPHYCop, is there any
>> need for any other LFB instance to be per-mediaport? Indeed, the reverse
>> case can be also argued and you have everything per-mediaport.
>> But we do have ability for grouping of LFB-ports which can be used to mux
>> to different instances of the same LFB class etc.
>> Figure1 seems to have have some LFBs FE wide and some mediaport
>> specific (the mediaencap seems to be port specific).
>>
>> cheers,
>> jamal
>>
>.

---

From: Jamal Hadi Salim <[email-masked]>
Date: Wed, 13 Apr 2011 06:56:26 -0400

I understand its value as metadata router. But why is it
necessary in figure1?
Could you not have an LFB-port-group at the EtherEncap to select
the egress port EtherMACOut?

Figure 1 is not consistent in the sense some LFBs are shown as
FE-wide instances and in others as mediaport-specific (refer to my
email to Joel). And then you have BasicMetadataDispatch routing
from FE-wide to mediaport-specifics..

cheers,
jamal

---

From: "Chuanhuang Li" <[email-masked]>
Date: Wed, 13 Apr 2011 19:25:58 +0800

Hi,Jamal

Joel and Weiming had discussed About why using a separated dispatch.
Please see the email:
subject: Re: Definition of EtherEncapsulator LFB
time: 2010-10-12 and 2010-10-13 

Yours,
Chuanhuang

======= 2011-04-13 18:49:37 Jamal Hadi Salim, wrote: =======

---

From: Jamal Hadi Salim <[email-masked]>
Date: Thu, 14 Apr 2011 07:27:26 -0400

Ok, I think i followed the gist of the conversation and i am
afraid I will have to revive some of that discussion. I apologize.

I need to think a little and see how best to express my view.

cheers,
jamal

---

From: Jamal Hadi Salim <[email-masked]>
Date: Thu, 14 Apr 2011 08:22:57 -0400

So my take on this is there is confusion at least for me because we
talk about L2 but dont go further (given it is out of scope). I believe we
need to talk about the relationship in the draft. This applies to
tunneling like GRE or ethernet MPLS etc not just to VLANs.

I have attached a text file with two figures. The first one shows
a small change to the egress side to be aware of L2
as we do in Ethermacin. The second model is better IMO.

Note: in both cases, there is no dispatch LFB, we have each
Etherencap using an LFB-port-group to select amongst many
EtherMAC outs.

cheers,
jamal

---

From: "Joel M. Halpern" <[email-masked]>
Date: Thu, 14 Apr 2011 10:52:45 -0400

I find the figures the way you labelled them confusing, because the 
EtherClassifier is not an L3 element.  In particular, for example, it 
would demux and hand off bridging control protocols.

The other confusing thing is that you are handing the local delivery 
determination to the bridging function.  I can live with that, but there 
are various things that have to be handled locally, and not handed to 
bridging logic.

Yours,
Joel

On 4/14/2011 8:22 AM, Jamal Hadi Salim wrote:
> On Thu, Apr 14, 2011 at 7:27 AM, Jamal Hadi Salim<[email-masked]>  wrote:
>>>>>> On Wed, Apr 13, 2011 at 5:32 AM, Jamal Hadi Salim<[email-masked]>  wrote:
>>>>>>

---

From: "Wang,Weiming" <[email-masked]>
Date: Fri, 15 Apr 2011 15:25:46 +0800

Jamal, 

We have already had discussions on why we should not reply on Etherencap LFB port groups  to do the port dispatch. The key point is we may have to consider some cases where logical port dispatch actually happens before Etherencap. This also actually means an Etherencap LFB we currently defined is capable of eithere per physical port per this instance or all physical ports one this encap LFB instance cases. We should not predetermine this relationship. 

Pls see emails on this topic in 
Sent: by Joel on Tuesday, October 12, 2010 10:41 PM
Subject: Re: Definition of EtherEncapsulator LFB

Sent: by weiming on Tuesday, October 12, 2010 8:20 PM
Subject: Re: Definition of EtherEncapsulator LFB

BTW, figure 1 in the draft is a global one FE-wide. But the example takes the case that only one Etherencap LFB instance for all physical ports. I think we can easierly draw an example where Etherencap LFB is instanciated per physical port. In this case, a  MetadataDispatch LFB should be applied after an NH LFB. Metadata used to dispatch will usually be logical port ID. Because of this, I think the NH LFB still needs to have logical port ID as its metadata output. 

thanks,
Weiming

----- Original Message ----- 
From: "Jamal Hadi Salim" <[email-masked]>
Subject: Re: [forces] BasicMetadataDispatch WAS(Re: LFB LPM issue

cheers,
jamal

>>>>>> specific (the mediaencap seems to be port specific).

--------------------------------------------------------------------------------

> _______________________________________________
> forces mailing list
> [email-masked]
> https://www.ietf.org/mailman/listinfo/forces
>

---

From: Jamal Hadi Salim <[email-masked]>
Date: Fri, 15 Apr 2011 07:29:52 -0400

Weiming,

On Fri, Apr 15, 2011 at 3:25 AM, Wang,Weiming <[email-masked]> wrote:
> Jamal,
>
> We have already had discussions on why we should not reply on Etherencap LFB port groups  to do the port dispatch.
>The key point is we may have to consider some cases where logical port dispatch actually happens before Etherencap.

That makes sense when a packet is coming from a host (CE/FE) being egressed, no?
Which that should fit in this model depending what you want to do. You
may want to inject it before
routing decision is made in which case decisions on how it proceeds
will be made starting at LPM
or you can bypass that and you have to provide the metadata etherencap expects.
Did i miss something?

>This also actually means an Etherencap LFB we currently defined is capable of eithere per physical port per
> this instance or all physical ports one this encap LFB instance cases. We should not predetermine this relationship.

IMO, thats a different issue.
a logical port on top of a physical port or another logical port
(infinitum) is well understood and supported by
the model. From current LFB, you need to know:
 i)  what the next LFB (instance) - solved by LFB-ports and you also need
ii) to pass some metadata the next LFB can use to select some table
entry to make its decision.

If you are trying to exemplify how one would model a L3 forwarding
path, then I think the metadata router
to avoid doing the above is an unnecessary optimization and more
importantly confusing.

> Subject: Re: Definition of EtherEncapsulator LFB

I just looked at those author discussions and i am still unsure as above.
Note: I already apologized for repeating some of these discussions ;->

>BTW, figure 1 in the draft is a global one FE-wide.
>But the example takes the case that only one Etherencap LFB instance for all physical ports.
>I think we can easierly draw an example where Etherencap LFB is instanciated per physical port.
>In this case, a  MetadataDispatch LFB should be applied after an NH LFB.
>Metadata used to dispatch will usually be logical port ID. Because of this, I think the NH LFB still
>needs to have logical port ID as its metadata output.
>

In order for the reader to follow the text it is important to provide
good context.
Maybe we need more than one example. Maybe we also need a big picture view
of how things fit like the diagram i posted...

cheers,
jamal

---

From: Jamal Hadi Salim <[email-masked]>
Date: Fri, 15 Apr 2011 07:38:09 -0400

On Thu, Apr 14, 2011 at 10:52 AM, Joel M. Halpern <[email-masked]> wrote:
> EtherClassifier is not an L3 element.  In particular, for example, it would
> demux and hand off bridging control protocols.

I struggled with that one - for that moment just wanted to put out a general
view of my thoughts but will revisit if get agreement from Weiming.

> determination to the bridging function.  I can live with that, but there are
> various things that have to be handled locally, and not handed to bridging
> logic.
>

That seemed like the easiest thing to do (an maybe a micro optimization); but
I have seen that model used in a few places - in those cases, the L2 table will
need to have a flag which indicates "for this (L2) table entry, you
need to have L3
processing".

cheers,
jamal

---

From: "Wang,Weiming" <[email-masked]>
Date: Sat, 16 Apr 2011 10:20:39 +0800

Jamal,

Re: Sorry for my previous text where "reply" should be "rely". What I tried to explain is, there are two possible use cases for Etherencap LFB. One is to mux all inputs and let one such LFB work  for all physical ports, just like the examples in current draft figure 1. Another possible case is, the Ethernet encap function is implemented in distributed line cards, which may even have to be modeled as one Etherencap LFB per physical port. For latter cases, after NH LFB where a output port ID is assigned, should be followed with a dispatch to route packets to individual output port, just like in reality a switch fabric does in a router architecture. 

Re: this problem is, if we do not define the metadata router as you said, how can we deal with the second case I mentioned above where a switch fabric like function is adopted. Moreover, even you try to avoid a separate metadata router LFB, you have to put the function of dispatch-by-metadata inside the Etherencap LFB. It just saves nothing.

Re: this can be done.

Maybe we need more than one example. Maybe we also need a big picture view
of how things fit like the diagram i posted...
Re: which another example do you think we better to show? IMO, I think the diagram you posted might be with two questions, 1) not FE-wide, only dipicts one physical port, so it is not a big picture view. 2) to show the L2 processing LFB explicitly in the figure while we actually does not define the LFB will result in confusing. 

thanks,
Weiming

---

From: Jamal Hadi Salim <[email-masked]>
Date: Sun, 17 Apr 2011 09:41:17 -0400

Hi Weiming,

On Fri, Apr 15, 2011 at 10:20 PM, Wang,Weiming <[email-masked]> wrote:

> Re: Sorry for my previous text where "reply" should be "rely". What I tried to explain >is, there are two possible use cases for Etherencap LFB. One is to mux all inputs >and let one such LFB work  for all physical ports, just like the examples in current >draft figure 1. Another possible case is, the Ethernet encap function is implemented >in distributed line cards, which may even have to be modeled as one Etherencap >LFB per physical port. For latter cases, after NH LFB where a output port ID is >assigned, should be followed with a dispatch to route packets to individual output >port, just like in reality a switch fabric does in a router architecture.

Thanks - I understand you better; and I am glad we are having this
conversation,
because I could never have figured out why you made that decision (me being
the reader of the document)!

It sounds to me like you are describing a problem that we MUST address in
in the text. I dont think this issue is specific to just distributed
line cards but
also even within a single card, distributed LFB chips which may have
intermediate
"interconnects". At the minimal we need to have diagrams or text describing
that those approaches work with the provided model.

To simplify the discussion and build up on what you said above:
lets assume the issue is specific to case of an FE making a forwarding decision
for a port that sits in a different FE. I now understand why the NH needs to
point to a logical port mentioned by Joel. Such a port could be an
interconnect port (assume it is an ethernet port for simplicity) to the egress
FE before the packet is sent out the real port. So it seems etherencap should
be per-port, on the egress port; meaning the NH LFB needs to have entry
LFB-port-group in addition to the logical port ID. This LFB-port-group will
have the same exact entry for implementations that assume
one-etherencap-per-system. Maybe the EncapOutputIndex would suffice.

One-etherencap-per-system can handle easily the per-port-etherencap by
having LFB-port-groups that point to per-port-ethermacout.

So in both cases, i am still missing the need for a dispatch LFB.

Side note:
It may be important to make the rules that logical port IDs globaly unique
per NE. This way i can reference a logical port id in FE1 while it may
reside in FE3. Or it may be changed down the LFB stream by some
interconnect technology - but we are agnostic to that at LFB level.
My only worry is that because this is generic a 32-bit space may be
insufficient for logical ports.

Since this is a long email - and i am hoping it doesnt go into TL;DR bucket
i will wait then respond in another email to the rest of your email later.
Chuanhuang, this is also why i am not responding to your other email.
I will catch up later with it.

cheers,
jamal

---

From: Jamal Hadi Salim <[email-masked]>
Date: Sun, 17 Apr 2011 10:18:19 -0400

part 2: After more coffee.

On Fri, Apr 15, 2011 at 10:20 PM, Wang,Weiming <[email-masked]> wrote:

>Re: this problem is, if we do not define the metadata router as you said, how can >we deal with the second case I mentioned above where a switch fabric like function >is adopted. Moreover, even you try to avoid a separate metadata router LFB, you >have to put the function of dispatch-by-metadata inside the Etherencap LFB. It just >saves nothing.
>

You may be right, it could be simpler to have a metadata router from a
high level description point of view. It may also be optimal.
I am just not sure how the model mandates it.

> Re: which another example do you think we better to show? IMO, I think the >diagram you posted might be with two questions, 1) not FE-wide, only dipicts one >physical port, so it is not a big picture view. 2) to show the L2 processing LFB >explicitly in the figure while we actually does not define the LFB will result in >confusing.

L2 processing is already mentioned in the draft and the example i gave indicates
port#N where N is an arbitrary number. So it is not specific to one
physical port.

cheers,
jamal

---

From: "Joel M. Halpern" <[email-masked]>
Date: Sun, 17 Apr 2011 11:39:06 -0400

We do, as you say, need to put better text in here.  Weiming will not be 
there to remind each reader of the discussion :-)

The scope of uniqueness of the port IDs is up to the CE.  It assigns 
them.  It builds the table entries.
While I tend to like NE-wide uniqueness of logical port IDs, and 
separation of logical port IDs from physical port IDs, at most this can 
be a suggested behavior.  After all, with a sufficiently (maybe 
excessively) clever CE, the system will work with duplication of IDs 
across FEs.  Even if one has a meta-data passing interface (which is not 
yet part of the spec.)

The general point is that one comes out of the NH LFB knowing the 
output.  But one could easily have additional common processing after 
taht, before the ether-encap.  (charging, filtering, other policy).  As 
such, there has to be enough meta-data to allow later steering.

Now, if one has meta-data to allow later steering, it is pretty useful 
to have an LFB which can do that steering.  We can either define one 
that is specific, or define a generic one.  (The generic has rather more 
complex information structure, but is MUCH more powerful.)  One could 
also define a generic as if it were, for now, a virtual LFB class and 
then subclass it immediately to the case that we want.  But I don't see 
much value in that.

Yours,
Joel

On 4/17/2011 10:18 AM, Jamal Hadi Salim wrote:
> On Fri, Apr 15, 2011 at 10:20 PM, Wang,Weiming<[email-masked]>  wrote:
>
>
>> Re: this problem is, if we do not define the metadata router as you said, how can>we deal with the second case I mentioned above where a switch fabric like function>is adopted. Moreover, even you try to avoid a separate metadata router LFB, you>have to put the function of dispatch-by-metadata inside the Etherencap LFB. It just>saves nothing.
>>
>
>> Re: which another example do you think we better to show? IMO, I think the>diagram you posted might be with two questions, 1) not FE-wide, only dipicts one>physical port, so it is not a big picture view. 2) to show the L2 processing LFB>explicitly in the figure while we actually does not define the LFB will result in>confusing.
>
>

---

From: Jamal Hadi Salim <[email-masked]>
Date: Sun, 17 Apr 2011 13:52:09 -0400

> there to remind each reader of the discussion :-)

Yes, that would help immensely.
Even pointing out where the possible "port fanout" points would be
sufficient justification to describe the need for the metadata router.
Figure 1 points to a fanout after EtherEncap.
Weiming pointed of one after NH. I am assuming it is very very uncommon
to find anything else in a router.
Do we need something in those LFBs to tell the CE that there is a fanout
that follows it? it does not seem sufficient for the FE to describe just the
LFB topology.

> The scope of uniqueness of the port IDs is up to the CE.  It assigns them.
>  It builds the table entries.
> While I tend to like NE-wide uniqueness of logical port IDs, and separation
> of logical port IDs from physical port IDs, at most this can be a suggested
> behavior.  After all, with a sufficiently (maybe excessively) clever CE, the
> system will work with duplication of IDs across FEs.  Even if one has a
> meta-data passing interface (which is not yet part of the spec.)
>

Ok. It really is upto the CE to mediate all that.

> The general point is that one comes out of the NH LFB knowing the output.
> But one could easily have additional common processing after taht, before
> the ether-encap.  (charging, filtering, other policy).  As such, there has
> to be enough meta-data to allow later steering.

And all that can be discovered (niceties of ForCES).

> Now, if one has meta-data to allow later steering, it is pretty useful to
> have an LFB which can do that steering.  We can either define one that is
> specific, or define a generic one.  (The generic has rather more complex
> information structure, but is MUCH more powerful.)  One could also define a
> generic as if it were, for now, a virtual LFB class and then subclass it
> immediately to the case that we want.  But I don't see much value in that.

It does seem like the current description is trying to be generic.
I think i will be fine with leaving the metadata router along with a good
description and insisting that figure 1 and at least the two other
possibilities are merely examples.

cheers,
jamal

---

From: "Joel M. Halpern" <[email-masked]>
Date: Sun, 17 Apr 2011 14:58:45 -0400

Clarifying one point below:

On 4/17/2011 1:52 PM, Jamal Hadi Salim wrote:
...
> that follows it? it does not seem sufficient for the FE to describe just the

Remember that the ForCES LFB includes the information as to what LFB 
pair sequences (A followed by B) are allowed.  So if the LFB requires 
that the etherencap or other media encap follow the NH LFB, then it can 
specify that and the CE will know it.

Yours,
Joel

---

From: Jamal Hadi Salim <[email-masked]>
Date: Sun, 17 Apr 2011 20:14:41 -0400

On Sun, Apr 17, 2011 at 2:58 PM, Joel M. Halpern <[email-masked]> wrote:

> Remember that the ForCES LFB includes the information as to what LFB pair
> sequences (A followed by B) are allowed.  So if the LFB requires that the
> etherencap or other media encap follow the NH LFB, then it can specify that
> and the CE will know it.
>

Makes sense - but something is still obscuring the view in my mind
especially after i tried to put together the attached slide to clarify
my understanding.
Let me know if this is the intent. If it is, could someone elucidate how
one would go about mapping as a NH port something that fits
"port x on FE y"?

cheers,
jamal

---

From: "Joel M. Halpern" <[email-masked]>
Date: Sun, 17 Apr 2011 20:19:18 -0400

I am not sur ehow the slide relates to the question.
Remember that the design is explicitly constrained.  We do not currently 
support passing metadata between FEs.  Therefore, there is no way to say 
'send this to some other specified FE, and preserve this metadata 
information for that FE.'  Clearly, we can do so later, if we want to.  
For now, you can only specify the output port as understood by this FE.

Note: That does not prohibit the CE from configuring the dispatch logic 
to know that logical ports 1-100 are actually physical port A, which is 
an inter-FE port.  All sorts of games the CE can play under the covers.

Yours,

Joel
On 4/17/2011 8:14 PM, Jamal Hadi Salim wrote:
> On Sun, Apr 17, 2011 at 2:58 PM, Joel M. Halpern<[email-masked]>  wrote:
>

---

From: Jamal Hadi Salim <[email-masked]>
Date: Sun, 17 Apr 2011 20:30:51 -0400

On Sun, Apr 17, 2011 at 8:19 PM, Joel M. Halpern <[email-masked]> wrote:
> information for that FE.'  Clearly, we can do so later, if we want to.  For
> now, you can only specify the output port as understood by this FE.
>
> Note: That does not prohibit the CE from configuring the dispatch logic to
> know that logical ports 1-100 are actually physical port A, which is an
> inter-FE port.  All sorts of games the CE can play under the covers.
>

ok.
Does the metadata router as i had it describe to the intended use?

cheers,
jamal

---

From: "Wang,Weiming" <[email-masked]>
Date: Mon, 18 Apr 2011 21:35:52 +0800

Jamal, 

I think the use of the metadata dispatch as you presented by the slide are what we intended.

While something I now can not follow is your description as below: 

● Etherencap uses the logical port id from NH to lookup a table and resolve:
   SrcMAC, dstMAC, VLAN (optional) and an output logical port
● Dstmac and port may be resolved by ARP using NHIP (out of scope)

Which is actually exactly what we do in current version.  As a result, what is the use of  NHEncapInfoIndex as we are now discussing recently considering that we'v used the logical port id as the lookup index in Etherencap? 

Another point that I can not follow your discusssion with Joel and Chuanhuang is, what is exactly the meaning "ARP being out of scope of this library"?  Shall we not define the ARP LFB or just we not describe the ARP function totally in the library document?

thanks,
Weiming

----- Original Message ----- 

ok.

cheers,
jamal

---

From: "Joel M. Halpern" <[email-masked]>
Date: Mon, 18 Apr 2011 09:42:21 -0400

The logical output port is not enough information to select the dst mac. 
  There is NOT a separate logical output port per neighbor.  We had be 
using the next-hop IP address, whcih has several limitations. 
Therefore, we use a new table index metadatum instead.

Yours,
Joel

On 4/18/2011 9:35 AM, Wang,Weiming wrote:
> From: "Jamal Hadi Salim"<[email-masked]>
>
> On Sun, Apr 17, 2011 at 8:19 PM, Joel M. Halpern<[email-masked]>  wrote:

---

From: Jamal Hadi Salim <[email-masked]>
Date: Mon, 18 Apr 2011 10:24:26 -0400

To followup on Joel:

Yes, I meant "some index to be used for lookup".
I find it confusing sometimes with the term  "logical port id "  because
the term "port" could be used to mean both an LFB port as well
as a port that is eventually going over some media.
Which is the correct meaning of  "logical port id " ?

Weiming: I didnt want to revisit the discussion with Chuanhuang yet;
i just wanted to settle on the big picture then we could go into details.
So if what i described is sufficient we could go into the details

cheers,
jamal
On Mon, Apr 18, 2011 at 9:42 AM, Joel M. Halpern <[email-masked]> wrote:
>  There is NOT a separate logical output port per neighbor.  We had be using
> the next-hop IP address, whcih has several limitations. Therefore, we use a
> new table index metadatum instead.
>
> Yours,
> Joel
>
>> Which is actually exactly what we do in current version.  As a result,
>> what is the use of  NHEncapInfoIndex as we are now discussing recently
>> considering that we'v used the logical port id as the lookup index in
>> Etherencap?
>>
>> Another point that I can not follow your discusssion with Joel and
>> Chuanhuang is, what is exactly the meaning "ARP being out of scope of this
>> library"?  Shall we not define the ARP LFB or just we not describe the ARP
>> function totally in the library document?
>>
>> thanks,
>> Weiming
>>
>>> to

---

From: "Joel M. Halpern" <[email-masked]>
Date: Mon, 18 Apr 2011 10:31:45 -0400

We have noted repeatedly that using port for the connections between 
LFBs was probably a mistake.  But we could not find a better term.

Which is why I try to remember to refer to the piece of port information 
produced as an output metadatum by the NH LFB as an "L3 logical port 
ID", i.e. it represents the single or bridged collection of physical / 
VLAN exits from the FE.

Yours,
Joel

On 4/18/2011 10:24 AM, Jamal Hadi Salim wrote:
> On Mon, Apr 18, 2011 at 9:42 AM, Joel M. Halpern<[email-masked]>  wrote:

---

From: Jamal Hadi Salim <[email-masked]>
Date: Mon, 18 Apr 2011 10:38:14 -0400

It would be useful if we fixed this doc to prefix the port with "LFB" when
that is what it is meant (readers of this document will be the type that
understand port to mean media-port).

cheers,
jamal

On Mon, Apr 18, 2011 at 10:31 AM, Joel M. Halpern <[email-masked]> wrote:
> We have noted repeatedly that using port for the connections between LFBs
> was probably a mistake.  But we could not find a better term.
>
> produced as an output metadatum by the NH LFB as an "L3 logical port ID",
> i.e. it represents the single or bridged collection of physical / VLAN exits
> from the FE.
>
> Yours,
> Joel
>
>>> using
>>>> I think the use of the metadata dispatch as you presented by the slide
>>>> are
>>>> what we intended.
>>>>
>>>>> For

---

From: "Wang,Weiming" <[email-masked]>
Date: Tue, 19 Apr 2011 11:06:51 +0800

Jamal and all,
 
 By following recent discussions, I and Chuanhuang are a little bothered by the status that it is hard for both us to allocate the discussions  into specific definitions of related LFBs, although some new thoughts have been presented, i.e., some thoughts are still too vague to be reflected in the modification of LFB definitions.
 
 Chuanhuang have propsed a modification on the modification of NH and Etherencap LFBs in message of 
 Sent: Thursday, April 14, 2011 10:43 PM
 Subject: Re: [forces] LFB LPM issue
 
 I do hope to get reply from this modifications, or pls just present your modifications of the difinitions as well as your thoughts.
 
 thanks,
Weiming

---

From: Jamal Hadi Salim <[email-masked]>
Date: Tue, 19 Apr 2011 08:51:34 -0400

Weiming,

I will do writeup to summarize things hopefully by end of today my time
now that we seem to all agree on the big picture at least.

cheers,
jamal