Subject: [Dcrouting] draft-przygienda-rift-03

From: Robert Raszuk <[email-masked]>
Date: Thu, 11 Jan 2018 10:30:00 +0100

Hi,

I have one little question/doubt on scalability point of RIFT ...

Assume that someone would like to signal IPv6 prefix SID for Segment
Routing in the underlay within RIFT.

Wouldn't it result in amount of protocol state in full analogy to massive
deaggregation - which as of today is designed to be very careful and
limited operation only at moments of failure(s) ?

I sort of find it a bit surprising that RIFT draft does not provide
encoding for SID distribution when it is positioned as an alternative to
other protocols (IGPs or BGP) which already provide ability to carry all
types of SIDs.

Cheers,
Robert.

PS1: Horizontal links which were discussed could be installed to offload
from fabric transit massive amount of data (ex: storage mirroring) directly
between leafs or L3 TORs and not to be treated as "backup".

PS2: Restricting any protocol to specific topologies seems like pretty
slippery slope to me. In any case if protocol does that it should also
contain self detection mechanism of "unsupported topology" and flash red
light in any NOC.

---

From: Tony Przygienda <[email-masked]>
Date: Thu, 11 Jan 2018 09:40:04 -0800

Robert, productive points, thanks for raising them ... I go a bit in depth

1. I saw no _real_ use-cases for SID in DC so far to be frank (once you run
RIFT). The only one that comes up regularly is egress engineering and that
IMO is equivalent to SID=leaf address (which could be a HV address of
course once you have RIFT all way down to server) so really, what's the
point to have a SID? It's probably much smarter to use IBGP & so on overlay
to do this kind of synchronization if needed since labels/SIDs become very
useful in overlay to distinguish lots stuff there like VPNs/services which
you'd carry e.g. in MPLSoUDP. In underlay just use the destination v4/v6
address. Having said that, discussion always to be had if you pay me dinner
;--) and I know _how_ we can do SIDs in RIFT since I thought it through but
again, no _real_ use case so far. And if your only concern is to "shape
towards a prefix" we have PGP in the draft which doesn't need new silicon
;-P And then ultimately, yes, if you really, really want a SID per prefix
everywhere then you'll carry  SIDs to everywhere since unicast SIDs are
really just a glorified way to say "I have this non-aggreagable 20 bit IP
host address" which architecturally is a very interesting proposition in
terms of scaling (but then again, no account for taste and RFC1925 clause 3
applies) ...  Your LSDB will be still much smaller, your SPF will be still
simple on leaf in RIFT but your FIB will blow up and anything changing on a
leaf shakes all other leafs (unless you start to run pollicies to control
distribution @ which point in time you start to baby-sit your fabric @ high
OPEX). One of the reasons to do per-prefix SID would be non-ECMP anycast
(where SIDs _are_ in fact usefull) but if you read RIFT draft carefully you
will observe that RIFT can do anycast without need for ECMP, i.e. true
anycast in a sense and with that having anycast SID serves no real purpose
in RIFT and is actually generally much harder to do since you need globally
unique label blocks and so on ...

2. Horizontal links on CLOSes are not used that way normally all I saw
since your blocking goes to hell unless you provision some kind of really
massive parallel links between ToRs _and_ understand your load. We _could_
build RIFT that way but you give up balancing through the fabric and
loop-free property in a sense (that's a longish discussion  and scaling
since now you have prefixes showing up all kind of crazy places instead of
default). I see enough demand, we get there ...  Otherwise RFC1925 clause
10 and 5.

3. PS1: Yes, lots of things "could" be done and then we "could" build a
protocol to do that and RFC1925 clause 7 and 8 applies. Such horizontal
links, unless provisioned correctly will pretty much just ruin your
blocking/loss on the fabric is the experience (which the math supports). In
a sense if you know your big flows you can build a specialized topology to
do the optimal distribution (MPLS tunnels anyone ;-) but the point of
fabric is that it's a fabric (i.e. load agnostic, cheap, no OPEX and easily
scalable). Otherwise a good analogy would be that you like to build special
RAM chips for the type of data structures you are storing and we know how
well that scales over time. We know now that within 3-4 years
characteristics of DC flows flip upside down without a sweat when people go
from server/client to microservices, from servers to containers and so on
and so on. So if you can't predict your load all the time you need a
_regular_ topology where _regular_ is more of a mathematical than a
protocol discussion. Fabric analogy of "buy more RAM chips in Fry's and
just stick them in" applies here. So RIFT is done largely to serve a
well-known structure called a "lattice" (with some restrictions) since we
need an "up" and "down". Things like hypercubes, thoroidal meshes and so on
and so on exist but CLOS won for a very good reason in history for that
kind of problems (once you move to NUMA other things win ;-) And if you
know your loads and your can heft the OPEX and you like to play with
protocols generally and if you can support the scale in terms of leaf FIB
sizes, flooding, slower convergence & so on & so on and you run flat IGP on
some kind of stuff that you build that doesn't even have to be regular in
any sense. We spent many years solving THAT problem obviously and doing
something like RIFT to replace normal IGP is of limited interest IMO
(albeit certain aspects having to do with modern implemenation techniques
may get us there one day but it's much less of pressing problem than
solving specialized DC routing well IMO again).

3. PS2: RIFT cannot build an "unsupported topology" no matter how you cable
(that's the point of it) or rather we have miscabling detection and do not
form adjacencies when you read the draft carefully. That's your "flash red
light" and it comes included for free with my compliments  ;-) ...
Otherwise RFC1925 clause 10.

Otherwise, if you have concrete charter points you'd like to add, be more
specific in your asks and we see what the list thinks after ...

thanks

--- tony

On Thu, Jan 11, 2018 at 1:30 AM, Robert Raszuk <[email-masked]> wrote:

> _______________________________________________
> Dcrouting mailing list
> [email-masked]
> https://www.ietf.org/mailman/listinfo/dcrouting
>
>

---

From: Tony Przygienda <[email-masked]>
Date: Thu, 11 Jan 2018 10:13:38 -0800

Having said that there are interesting use cases we talk of "some binding
per node" but that's more of a KV store corner where people think it's very
useful to have the spines pushing stuff down to all nodes or (at cost of
stability) having a node push some stuff up that gets pushed down to all
other nodes. That's more of a "per leaf node" information case and pretty
good unless leafs go crazy doing dynamic re-assignment (but we can dampen
that in spines @ every level if needed) but it's not _per prefix_ which you
seemed to ask about ....

--- tony

On Thu, Jan 11, 2018 at 9:40 AM, Tony Przygienda <[email-masked]>
wrote:

> 1. I saw no _real_ use-cases for SID in DC so far to be frank (once you
> run RIFT). The only one that comes up regularly is egress engineering and
> that IMO is equivalent to SID=leaf address (which could be a HV address of
> 3. PS2: RIFT cannot build an "unsupported topology" no matter how you
> cable (that's the point of it) or rather we have miscabling detection and
> do not form adjacencies when you read the draft carefully. That's your
> "flash red light" and it comes included for free with my compliments  ;-)
> ... Otherwise RFC1925 clause 10.
>

---

From: Robert Raszuk <[email-masked]>
Date: Thu, 11 Jan 2018 19:54:07 +0100

Hi Tony,

Thx for elaborating ...

Two small comments:

A) SID/SR use case in underlay could be as simple as gracefully taking a
fabric node out of service. Not much OPEX needed if your NMS is decent.
Otherwise in normal link state I can do overload bit, in BGP number of
solutions from shutdown to MED to LP ... depending what is your BGP design.
In RIFT how do you do that ? Note that overlay (if such exist) does not
help here.

B) For horizontal links imagine you have servers with 40 GB ports to TOR.
Then you have Nx100 GB from TOR up. You are going to oversubscribe on TOR
(servers to fabric) most likely 2:1 .. 3:1 etc. So if I want to
interconnect TORs because I do know that servers behind those TORs need to
talk to each other in a non blocking fashion _and_ I have spare 100 GB
ports on TORs having routing protocol which does not allow me to do that
seems pretty limited - wouldn't you agree ?

Thx,
R.

---

From: Tony Przygienda <[email-masked]>
Date: Thu, 11 Jan 2018 13:19:20 -0800

A. Reading the draft you'll find the overload bit on RIFT which IMO is the
best out-of-production solution.
B. Depends what your LEAF is. If LEAF is your TOR (most stuff today) then
RIFT will do that just fine using LEAF2LEAF procedures in fact. If we start
to run RIFT @ the server level then we'd need discuss but people do not do
horizontal links in non-leaf-to-non-leaf as far I saw (except protection
here and there) but then again, they mostly don't extend the fabric routing
protocol into the server (which RIFT intends to do). So, for today you're
good. Once you want RIFT on server I think having spine horizontals is not
a good idea in general

I hope that computes and I'd encourage you to read the draft again, maybe
without the "it should work just like BGP or just like today's IGP" hat on
...  ;-)  Of course, paying me fancy dinners and asking questions, letting
me ramble then is a valid substitute ;-)

--- tony

On Thu, Jan 11, 2018 at 10:54 AM, Robert Raszuk <[email-masked]> wrote:

>>> Wouldn't it result in amount of protocol state in full analogy to
>>> massive deaggregation - which as of today is designed to be very careful
>>> and limited operation only at moments of failure(s) ?
>>>

---

From: Tony Przygienda <[email-masked]>
Date: Sat, 13 Jan 2018 10:16:48 -0800

So I thought over your horizontal link case to shortcut the spine levels
for some kind of traffic again and I think the current draft actually
covers that if you're willing to provision things correctly yourself via
S-PGP.
Let me see whether we agree on this picture first:

.                +--------+          +--------+
.                |        |          |        |          ^ N
.                |Spine 21|          |Spine 22|          |
.Level 2         ++-+--+-++          ++-+--+-++        <-*-> E/W
.                 | |  | |            | |  | |           |
.             P111/2|  |P121          | |  | |         S v
.                 ^ ^  ^ ^            | |  | |
.                 | |  | |            | |  | |
.  +--------------+ |  +-----------+  | |  | +---------------+
.  |                |    |         |  | |  |                 |
. South +-----------------------------+ |  |                 ^
.  |    |           |    |         |    |  |              All TIEs
.  0/0  0/0        0/0   +-----------------------------+     |
.  v    v           v              |    |  |           |     |
.  |    |           +-+    +<-0/0----------+           |     |
.  |    |             |    |       |    |              |     |
.+-+----++ optional +-+----++     ++----+-+           ++-----++
.|       | E/W link |       +=====+       |           |       |
.|Node111+----------+Node112|     |Node121|           |Node122|
.+-+---+-+          ++----+-+     +-+---+-+           ++---+--+
.  |   |             |   South      |   |              |   |
.  |   +---0/0--->-----+ 0/0        |   +----------------+ |
. 0/0                | |  |         |                  | | |
.  |   +---<-0/0-----+ |  v         |   +--------------+ | |
.  v   |               |  |         |   |                | |
.+-+---+-+          +--+--+-+     +-+---+-+          +---+-+-+
.|       |  (L2L)   |       |     |       |  Level 0 |       |
.|Leaf111~~~~~~~~~~~~Leaf112|     |Leaf121|          |Leaf122|
.+-+-----+          +-+---+-+     +--+--+-+          +-+-----+
.  +                  +    \        /   +              +
.  Prefix111   Prefix112    \      /   Prefix121    Prefix122
.                          multi-homed
.                            Prefix
.+---------- Pod 1 ---------+     +---------- Pod 2 ---------+

I assume here that what you ask for is the following scenario:
a) POD1 being a compute generating very heavy load towards storage in
   Prefix121.
b) traffic from POD1 NOT being balanced through the spines but taking
   a horizontal link Node112 to Node121 to reach your storage in Prefix121
   to save bandwidth? or delay?
c) The key to riches is -04 section4.2.5.1
<https://tools.ietf.org/html/draft-przygienda-rift-04#section-4.2.5.1>.
Northbound SPF

  in the paragraph "Other south prefixes found when crossing E-W link MAY
be used IIF". Now, we could make it a MUST (but it's really an
implementation knob IMO) and what it says is that if you are willing to
inject an "S-PGP" @ Node121 for Prefix121 it will get flooded to Node112
and Node112 will have a more specific match than the default in N-SPF. From
121 normal RIFT takes over since the normal N-Prefix for Leaf121 kicks in
on Node121. I assume Node112 policy on the ingress is to not propagate the
S-PGP south but use it for N-SPF only.

Observe that

a) you have to switch off miscabling detection for PoD# on those nodes
since you are "crossing PoDs illegally"

b) if you want whole Pod#1 to do that you either cable Node111 to Node121
as well (which will load balance whole Pod#1 towards storage without using
Spine)  OR you propagate S-PGPs south towards leafs (which will cost you
leaf FIB of course but make sure ALL traffic to storage goes over Node112
only).

c) Your forwarding on Node112 can actually even go haywire & load-balance
some of the traffic to Prefix121 using this S-PGP over Node121 and some
still using the default route towards spine(which here is a blatant
violation of LPM of course) and RIFT will work just fine (unless you loop
yourself to death with PGPs you install) but that of course is a deep
rathole in itself. I just mention it to show why the "non-looping" design
is so important and makes for the shortcomings for the SPF on a fabric,
predicted by the non-directional mesh property that Dijkstra solved in his
time.

so?

--- tony

---

From: Robert Raszuk <[email-masked]>
Date: Sat, 13 Jan 2018 20:21:34 +0100

Hi Tony,

> ​
if you're willing to provision things correctly yourself via S-PGP.

I am not willing to do that. I would like routing to do it for me
auto-magically. But yes you got the question right. I was asking for
shortcuts between last levels of fabric. Not so much between Node 111 &
Node 112 - but you said it is optional so ok.

Side note: PGP abbrev. for vast majority of people means completely
different thing then what you defined it to mean locally in your draft. I
highly recommended you rename it in -05 version to PGD (policy guided
destination(s) or PGR (policy guided reachability/routing).

Now requirement to switch off miscabling detection is not acceptable. You
are stating that only rift knows how should I cable my fabric ? And if I
cable it some other way it will detect it as miscabling ? I think in most
cases of correct cables check you actually define the intent then network
detects if such intent is met.

> Node112 can actually even go haywire

That may be not best property of a routing protocol :)

Best,
R.

On Sat, Jan 13, 2018 at 7:16 PM, Tony Przygienda <[email-masked]>
wrote:

> covers that
> ​​
> if you're willing to provision things correctly yourself via S-PGP.
>>> Otherwise, if you have concrete charter points you'd like to add, be
>>> more specific in your asks and we see what the list thinks after ...
>>>
>>> thanks
>>>
>>> --- tony
>>>
>>>
>>>> PS1: Horizontal links which were discussed could be installed to
>>>> offload from fabric transit massive amount of data (ex: storage mirroring)
>>>> directly between leafs or L3 TORs and not to be treated as "backup".
>>>>

---

From: Tony Przygienda <[email-masked]>
Date: Sat, 13 Jan 2018 12:05:42 -0800

On Sat, Jan 13, 2018 at 11:21 AM, Robert Raszuk <[email-masked]> wrote:

> auto-magically.

The price of that would be that you can run strict SPF only then and loose
all the non-equal cost forwarding, including true anycast. Plus, once you
think that through in general sense you basically end up with host
addresses everywhere (since if you use aggregates you'll blackhole on link
failures). If you want to avoid SPF and host routes while still asking for
all the things you ask, then as a very experienced and highly regarded
architect here puts it to end such discussions "your problem is overly
constrained" and there is no protocol/algebra whatsoever that will give you
that ...

> But yes you got the question right. I was asking for shortcuts between
> last levels of fabric. Not so much between Node 111 & Node 112 - but you
> said it is optional so ok.
>

Sigh, I warned when the acronym was picked.  Come up with something that is
a word, the funnier the better, easier to remember for our brains that way.
Can be 4 letters ;-)  and will get you an Ack mention l;-)

> are stating that only rift knows how should I cable my fabric ?
>

No, it doesn't. You asked in your previous email for "red lights" when
topology is built incorrectly and you have it. Now you don't want it. Both
states cannot be true @ the same time unless you are entangled ;-)

> That may be not best property of a routing protocol :)
"haywire" is the wrong word. Rather,  I should have said "it will even
allow that". Now, why would you want the same src/dst pair sometimes
shortcut on the horizontal link and sometimes go up to spine I can't
imagine but you ask for all kind of very non-obvious things you seem to
like so I'm just pointing the space of possibilities out to you ...

-- tony

---

From: Robert Raszuk <[email-masked]>
Date: Sat, 13 Jan 2018 22:37:30 +0100

Tony,

> The price of that would be that you can run strict SPF only then and
loose all the non-equal cost forwarding, including true anycast.

1. Did anyone really asked that we should refrain from running "strict SPF"
in DCs ? Contrary I see group of folks keen on running it based on BGP-LS
like feeds :)

2. Is non-ECMP fwd-ing a feature or a bug in DC fabric ? When I described
to customers that with MPLS you can run non ECMP load balancing it was
shock .. but in WAN there are some use cases for it. But in DC fabric - I
doubt it.

3. True anycast is clearly possible with native link state and native
distance vector. Why the new "hybrid" glue of both would not allow it ?
Because it counts on massive aggregation - right ?

> Plus, once you think that through in general sense you basically end up
with host addresses everywhere (since if you use aggregates you'll
blackhole on link failures). If you want to avoid SPF and host routes

Host routes are just leaves so they hang off nodes and do not make SPF any
harder. Besides I am not convinced that building flat underlays with 1M
host routes is the right architecture - even if some folks ask you about
it.

Best,
R.

On Sat, Jan 13, 2018 at 9:05 PM, Tony Przygienda <[email-masked]>
wrote:

> Sigh, I warned when the acronym was picked.  Come up with something that
> is a word, the funnier the better, easier to remember for our brains that
> way. Can be 4 letters ;-)  and will get you an Ack mention l;-)
>
>
>>