Subject: [forces] Comments, questions: draft−joachimpillai−forces−interfelfb−01

From: Dave Hood <[email-masked]>
Date: Sun, 17 Mar 2013 16:38:03 +0000

The inter-ASIC ploy is a pretty blatant workaround for the single-box restriction. Is that regarded as kosher? Once we need to cross a physical boundary, be it between chips, blades, boxes or networks, surely we ought to be up-front about it and admit what we’re doing.

It is not clear from the draft whether each LFB link between two devices would have its own instance pair of inter-FELBs, or whether the identifiers are intended to act as multiplex tags, such that any number of [like?] LFBs on the same two devices could share an interface. I presume the latter, or perhaps the latter but with clarifications or restrictions, but a definite statement would be helpful.

When we start talking about network virtualization, we could end up with several layers [wholesale case, providers’ provider] of encapsulated metadata crossing wide geographic areas amongst a number of boxes. Maybe the virtual network planner will need to understand the topology-metadata relationships to ensure that little to no metadata is required? Even so, this draft does not allow for null metadata. Has all of that already been recognized and discussed? Or if this is out of scope, how would we ensure that it remains out of scope?

Dave

---

From: Jamal Hadi Salim <[email-masked]>
Date: Mon, 18 Mar 2013 08:08:13 -0400

Dave,

Your input is appreciated.

On Sun, Mar 17, 2013 at 12:38 PM, Dave Hood <[email-masked]> wrote:
> The inter-ASIC ploy is a pretty blatant workaround for the single-box
> restriction. Is that regarded as kosher?
> Once we need to cross a physical
> boundary, be it between chips, blades, boxes or networks, surely we ought to
> be up-front about it and admit what we’re doing.
>

Did you mean the text around figure2? Didnt grok the context.
We were hoping to support existing infrastructure with the transport
mapping (eg a marvel chip being able to transpose the metadata
across two chips/boards where sensible).

> It is not clear from the draft whether each LFB link between two devices
> would have its own instance pair of inter-FELBs, or whether the identifiers
> are intended to act as multiplex tags, such that any number of [like?] LFBs
> on the same two devices could share an interface. I presume the latter, or
> perhaps the latter but with clarifications or restrictions, but a definite
> statement would be helpful.

Agreed we need more illustrative text.
In current prototyping we use a pair of the inter-FE LFBs per FE and it seems
sufficient.
To be more general, however, the CE should have the luxury of
instantiating more
than one pair when needed. Example, it may be sensible and probably more
efficient (implementation dependent) in creating one pair per tenant.

> When we start talking about network virtualization, we could end up with
> several layers [wholesale case, providers’ provider] of encapsulated
> metadata crossing wide geographic areas amongst a number of boxes.

I am not familiar with the space you describe - is the anology equivalent
to say several encapsulated 8021q tags?

> Maybe the
> virtual network planner will need to understand the topology-metadata
> relationships to ensure that little to no metadata is required?

Not sure if i am answering the question:
The metadata is supposed to be described on a per-LFB basis, will be
documented on each LFB description document, created when needed
by the datapath and is to be consumed by downstream LFBs that
understand and expect it (per those LFB-specific documents).

The draft suggests that the CE will tell the inter-FE LFB of white or
black lists of metadata to pass through.

>Even so,
> this draft does not allow for null metadata.

Metadata is optional as is packet data (i.e one could send across only metadata)

>Has all of that already been
> recognized and discussed? Or if this is out of scope, how would we ensure
> that it remains out of scope?
>

It is within scope.
The current ForCES architecture allows either the FE or CE to send
optional packetdata + metadata (refer to:
https://tools.ietf.org/html/rfc5810#section-7.9 )
In the draft we used/assumed that experience. Do we need more
descriptive text?

cheers,
jamal

>
>
> Dave
>
>
> _______________________________________________
> forces mailing list
> [email-masked]
> https://www.ietf.org/mailman/listinfo/forces
>

---

From: "Joachimpillai, Damascene M" <[email-masked]>
Date: Mon, 18 Mar 2013 11:12:15 -0400

Jamal,

It may beg the question what constitutes a LFB and what constitutes an FE. 
Is TCAM a LFB or an FE? 

I have been using a very loose definition in our implementation. 

Regards,
DJ

-----Original Message-----
From: [email-masked] [mailto:[email-masked]] On Behalf Of Jamal Hadi Salim
Sent: Monday, March 18, 2013 8:08 AM
To: Dave Hood
Cc: [email-masked]
Subject: Re: [forces] Comments, questions: draft-joachimpillai-forces-interfelfb-01

Dave,

On Sun, Mar 17, 2013 at 12:38 PM, Dave Hood <[email-masked]> wrote:
> boundary, be it between chips, blades, boxes or networks, surely we 
> ought to be up-front about it and admit what we're doing.
>

Did you mean the text around figure2? Didnt grok the context.
We were hoping to support existing infrastructure with the transport mapping (eg a marvel chip being able to transpose the metadata across two chips/boards where sensible).

> It is not clear from the draft whether each LFB link between two 
> devices would have its own instance pair of inter-FELBs, or whether 
> the identifiers are intended to act as multiplex tags, such that any 
> number of [like?] LFBs on the same two devices could share an 
> interface. I presume the latter, or perhaps the latter but with 
> clarifications or restrictions, but a definite statement would be helpful.

Agreed we need more illustrative text.
In current prototyping we use a pair of the inter-FE LFBs per FE and it seems sufficient.
To be more general, however, the CE should have the luxury of instantiating more than one pair when needed. Example, it may be sensible and probably more efficient (implementation dependent) in creating one pair per tenant.

> When we start talking about network virtualization, we could end up 
> with several layers [wholesale case, providers' provider] of 
> encapsulated metadata crossing wide geographic areas amongst a number of boxes.

I am not familiar with the space you describe - is the anology equivalent to say several encapsulated 8021q tags?

Not sure if i am answering the question:
The metadata is supposed to be described on a per-LFB basis, will be documented on each LFB description document, created when needed by the datapath and is to be consumed by downstream LFBs that understand and expect it (per those LFB-specific documents).

The draft suggests that the CE will tell the inter-FE LFB of white or black lists of metadata to pass through.

> recognized and discussed? Or if this is out of scope, how would we 
>ensure  that it remains out of scope?
>

It is within scope.
The current ForCES architecture allows either the FE or CE to send optional packetdata + metadata (refer to:
https://tools.ietf.org/html/rfc5810#section-7.9 ) In the draft we used/assumed that experience. Do we need more descriptive text?

cheers,
jamal

---

From: Joel <[email-masked]>
Date: Mon, 18 Mar 2013 11:18:50 -0400

The point I drew from Dave's message was a differet one.
hat we want to accomplish is to allow metadata to be passed between FEs, 
typically across Ethernet.  We should say that up front, and not try to 
justify it on the basis of specific hardware architecture (te scope of 
ASICs.)

The definition of an FE is effectively "the set of things under the 
control of a single client."  Conceptually, it is up to the FE itself 
how it passes metadata among its parts.  Where we get to talk about real 
bits on (real or virtual) wires is where things go in and out of an FE.

Yours,
Joel

---

From: Jamal Hadi Salim <[email-masked]>
Date: Wed, 20 Mar 2013 07:44:35 -0400

Hi DJ,
Sorry - very distracted elsewhere the last 2 days.

> It may beg the question what constitutes a LFB and what constitutes an FE.

I think it could be modelled as an LFB Class. And if you have multiple of
them, there would be multiple instances. Of course such an LFB class
could be in its own FE.

cheers,
jamal

---

From: Jamal Hadi Salim <[email-masked]>
Date: Wed, 20 Mar 2013 08:02:52 -0400

On Mon, Mar 18, 2013 at 11:18 AM, Joel <[email-masked]> wrote:
> what we want to accomplish is to allow metadata to be passed
> between FEs,

What you described is precisely the short term goal we want to
achieve. But i dont think we can ignore other FE interconnects
in a general standard, can we?
If i have several broadcom chips interconnected via higig then
it is important to be able to model that so i can control and
configure attributes from the CE (when it makes sense to do so).
Is the suggestion we just focus on ethernet only for simplicity of
a first cut?
On my part, at least, I think that ability to go across some IP
infrastructure (when FEs are VMs across networks) is important.

> The definition of an FE is effectively "the set of things under
> the control
> of a single client."  Conceptually, it is up to the FE itself how it passes
> metadata among its parts.  Where we get to talk about real bits on
> (real or virtual) wires is where things go in and out of an FE.
>

Agreed.
To my point above: Going across hi-gig is things going in/out of
FEs

cheers,
jamal

---

From: Jamal Hadi Salim <[email-masked]>
Date: Wed, 20 Mar 2013 08:14:26 -0400

Hrm. This was sitting on my outbox for some reason; if you see
duplicates ignore one.

Hi DJ,

> It may beg the question what constitutes a LFB and what constitutes an FE.

I think it could be modelled as an LFB Class. And if you have multiple
of them, there would
be multiple instances. Of course such an LFB class could be in its own FE.

cheers,
jamal

---

From: Dave Hood <[email-masked]>
Date: Wed, 20 Mar 2013 15:26:59 +0000

When two chips reside on the same board, it may suffice to define metadata forwarding in an abstract way; the chip designer and the board designer care how it happens, but standardization probably does not. When the chips reside on separate boards, but in a common backplane, this is arguably still true.

When two separate boxes are involved, and if they are expected to be interoperable, we surely need standard definitions of the metadata and how it is to be forwarded. Ethernet would be a logical starting point, but not necessarily the only encapsulation that needs to be defined.

Dave

-----Original Message-----
From: Jamal Hadi Salim [mailto:[email-masked]] 
Sent: Wednesday, March 20, 2013 5:03 AM
To: Joel
Cc: Joachimpillai, Damascene M; Dave Hood; [email-masked]
Subject: Re: [forces] Comments, questions: draft-joachimpillai-forces-interfelfb-01

On Mon, Mar 18, 2013 at 11:18 AM, Joel <[email-masked]> wrote:
> what we want to accomplish is to allow metadata to be passed between 
> FEs, typically across Ethernet.  We should say that up front, and not 
> try to justify it on the basis of specific hardware architecture (te 
> scope of
> ASICs.)
>

What you described is precisely the short term goal we want to achieve. But i dont think we can ignore other FE interconnects in a general standard, can we?
If i have several broadcom chips interconnected via higig then it is important to be able to model that so i can control and configure attributes from the CE (when it makes sense to do so).
Is the suggestion we just focus on ethernet only for simplicity of a first cut?
On my part, at least, I think that ability to go across some IP infrastructure (when FEs are VMs across networks) is important.

> how it passes metadata among its parts.  Where we get to talk about 
> real bits on (real or virtual) wires is where things go in and out of 
> an FE.
>

Agreed.
To my point above: Going across hi-gig is things going in/out of FEs

cheers,
jamal

---

From: Joel <[email-masked]>
Date: Wed, 20 Mar 2013 11:41:07 -0400

Where it gets more complicated than the picture below is when one looks 
at backplanes that are ATCA or similar (Ethernet, wit cards from 
different vendors), or chipmakers wanting to use standard interfaces in 
and out of their chip (e.g. Ethernet).

Having said that, I think it is much clearer if the document focuses on 
the inter-fe problem with separate boxes as the canonical case.  The 
other cases can be mention as furhter examples where this work may 
apply.  Doing it the other way looks like we are trying to slide a big 
deal under a hole in the existing limits.

Yours,
Joel

---

From: Jamal Hadi Salim <[email-masked]>
Date: Mon, 25 Mar 2013 09:19:38 -0400

On Wed, Mar 20, 2013 at 11:41 AM, Joel <[email-masked]> wrote:
> Where it gets more complicated than the picture below is when one looks at
> backplanes that are ATCA or similar (Ethernet, wit cards from different
> vendors), or chipmakers wanting to use standard interfaces in and out of
> their chip (e.g. Ethernet).
>

I think Joel states it well. Although ethernet could be looked at as
the universal
"backplane"; it is not the only one.
At minimal we need to mention the other possibilites. Note: ability to allow
an ecosystem of  chip/board/box vendors is/was the underlying driver for ForCES.
I should be able to buy a chip from Vendor X and plugin into my board using
their drivers and treat it as an LFB or even an FE. Likewise, i should be able
to buy a board from Vendor Y with network function A and plugin into my
NE. I should be able to buy a box from Vendor Z .. etc.

> Having said that, I think it is much clearer if the document focuses on the
> inter-fe problem with separate boxes as the canonical case.  The other cases
> can be mention as furhter examples where this work may apply.  Doing it the
> other way looks like we are trying to slide a big deal under a hole in the
> existing limits.
>

Board-to-board vs box-to-box should not be much different. I think if we focus
on ethernet and have one or two other docs describe other tunneling approaches
over IP satisfy the same requirements, we should be in good shape.

cheers,
jamal

---

From: Dave Hood <[email-masked]>
Date: Mon, 25 Mar 2013 13:59:15 +0000

Agreed. Board-board or box-box is a side issue; the point is to be able to tunnel, and Ethernet is a good start.
Dave

-----Original Message-----
From: Jamal Hadi Salim [mailto:[email-masked]] 
Sent: Monday, March 25, 2013 6:20 AM
To: Joel
Cc: Dave Hood; Joachimpillai, Damascene M; [email-masked]
Subject: Re: [forces] Comments, questions: draft-joachimpillai-forces-interfelfb-01

On Wed, Mar 20, 2013 at 11:41 AM, Joel <[email-masked]> wrote:
> Where it gets more complicated than the picture below is when one 
> looks at backplanes that are ATCA or similar (Ethernet, wit cards from 
> different vendors), or chipmakers wanting to use standard interfaces 
> in and out of their chip (e.g. Ethernet).
>

I think Joel states it well. Although ethernet could be looked at as the universal "backplane"; it is not the only one.
At minimal we need to mention the other possibilites. Note: ability to allow an ecosystem of  chip/board/box vendors is/was the underlying driver for ForCES.
I should be able to buy a chip from Vendor X and plugin into my board using their drivers and treat it as an LFB or even an FE. Likewise, i should be able to buy a board from Vendor Y with network function A and plugin into my NE. I should be able to buy a box from Vendor Z .. etc.

> Having said that, I think it is much clearer if the document focuses 
> on the inter-fe problem with separate boxes as the canonical case.  
> The other cases can be mention as furhter examples where this work may 

Board-to-board vs box-to-box should not be much different. I think if we focus on ethernet and have one or two other docs describe other tunneling approaches over IP satisfy the same requirements, we should be in good shape.

cheers,
jamal