Subject: [i2rs] draft-ietf-i2rs-problem-statement-01

From: Dean Bogdanovic <[email-masked]>
Date: Tue, 6 May 2014 23:05:01 +0000

Alia, Tom, Dave,

Have few comments on chapter 5 in the draft

The following requirement is fine, but some consideration have to be put in. Example, creating a filter construct or routing table has to be finished, before the next operation like add term to the filter or add route to the routing table is possible.

Multiple Simultaneous Asynchronous Operations:   A single application
      should be able to send multiple operations via I2RS without being
      required to wait for each to complete before sending the next.

So would suggest changing from multiple operations to multiple independent operations

High-throughput is always nice to have, but the quantification is a bit wishy washy.

 High-Throughput:   At a minimum, the I2RS Agent and associated router
      should be able to handle a considerable number of operations per
      second above what basic Netconf or a propretiary CLI can.

What does mean basic NETCONF? Today I can do couple hundred RPCs/sec over NETCONF. Question: what is RPC doing? Is it simple operational get or large config push? Or some other complex RPC?
There is also difference in creating logical interfaces and adding term to a filter or adding route to a routing table. Creating interface is most taxing one and adding term is fastest create operation. IMO, NETCONF has no problem with throughput, it is the instrumentation and that is something we should focus in I2RS.
What constructs do we want to control via I2RS and depending on that, make sure that the protocol can sustain max throughput.

In the next chapter you are mentioning

Responsive:   It should be possible to complete simple operations
      within a sub-second time-scale.

Here you are stating simple operation, so why not define few simple operations and put some quantification with it (in the below examples I'm pulling the numbers out of my hat)

example:

read routes - single prefix per transaction < 0.5sec
- multiple prefixes per transaction < time on the wire + 0.5sec
add route - single prefix per transaction < 0.5 sec
- multiple prefixes per transaction 1000 routes/sec

get filter info - single filter per transaction < 0.5sec
- multiple filters and terms per transaction  < time on the wire + 0.5sec
create filter - single filter per transaction < 0.5 sec
- multiple filters per transaction  2000/sec
add term to filter - single term per transaction < 0.5sec
- multiple terms per transaction  3000/sec

get interface info  - single interface per transaction < 0.5 sec
- multiple interfaces per transaction < time on the wire + 0.5 sec
create interface - single interface per transaction < 0.5 sec
- multiple interfaces per transaction 100/sec

I'm thinking how can you do assurance for i2rs. It looks as it is covered in following paragraph

  Scalable, Filterable Information Access:  To extract information in a
      scalable fashion that is more easily used by applications, the
      ability to specify filtering constructs in an operation requesting
      data or requesting an asynchronous notification is very valuable.

but it is not spelled out.

Dean

---

From: Alia Atlas <[email-masked]>
Date: Tue, 6 May 2014 23:24:19 +0000

Dean,

Thanks for the thorough review!  I'm traveling so will take until Friday for a full response.

Alia

-------- Original message --------
From: Dean Bogdanovic <[email-masked]>
Date: 05/06/2014 6:05 PM (GMT-06:00)
To: Alia Atlas <[email-masked]>,Thomas Nadeau <[email-masked]>,[email-masked]
Cc: "<[email-masked]>" <[email-masked]>
Subject: draft-ietf-i2rs-problem-statement-01

Alia, Tom, Dave,

example:

Dean

---

From: Jamal Hadi Salim <[email-masked]>
Date: Wed, 7 May 2014 09:13:08 -0400

On Tue, May 6, 2014 at 7:05 PM, Dean Bogdanovic <[email-masked]> wrote:

> High-throughput is always nice to have,

Is it "nice to have" or "must have"?

>but the quantification is a bit wishy washy.
>

Agreed.

> What does mean basic NETCONF? Today I can do couple hundred RPCs/sec over
> NETCONF. Question: what is RPC doing? Is it simple operational get or large
> config push?
> Or some other complex RPC?
> There is also difference in creating logical interfaces and adding term to a
> filter or adding route to a routing table. Creating interface is most taxing
> one and adding term is fastest create operation.
>IMO, NETCONF has no problem
> with throughput, it is the instrumentation and that is something we should
> focus in I2RS.
> What constructs do we want to control via I2RS and depending on that, make
> sure that the protocol can sustain max throughput.
>

We need to pick something as a metric since as you say it will depend on the
object type. The RIB table sounds like a reasonable object to standardize on.
Pick a number like 1M table entries (large enough so as to be measurable).

Throughput:
This refers to some bulk table create/updates/dumps. Very few transactions/sec
are needed, essentially in this case the agent and client are engaged
in batching the entries
they send to the other party.
To handwave some numbers performance numbers:
I am going to pick say 10s to 100s of thousands rows per/sec.
We need bidirectional numbers.

Transactions rate:
This is a measure of request/response. Again using the RIB example -
this would mean taking
each of the 1M rows and send them one at a time (i.e window of 1). Eg
a request from the
client to create/update is sent to the agent and when a success
response is received by the
client the next request is issued ..... until all of 1M transactions
are complete

> Here you are stating simple operation, so why not define few simple
> operations and put some quantification with it (in the below examples I'm
> pulling the numbers out of my hat)
>
> example:
>

We need to have some quantification numbers. They dont have to be precise
but could be in the ranges. I think what you describe above matches my thinking.
Your "multiple per transaction" is essentially a batching artifact
which falls under
"throughput" metric. The " single per transaction" is a transaction metric.

Having said that: I know you gave those numbers as examples, but they
are way too low ;->

> but it is not spelled out.

I would consider the throughput, transaction rate and latency in that
category as
well.

cheers,
jamal

---

From: Alia Atlas <[email-masked]>
Date: Fri, 16 May 2014 17:18:43 -0400

Hi Dean,

Sorry for the long delay :-(  I got felled by an ear infection.
Responses in-line.

On Tue, May 6, 2014 at 7:05 PM, Dean Bogdanovic <[email-masked]> wrote:
> The following requirement is fine, but some consideration have to be put in.
> Example, creating a filter construct or routing table has to be finished,
> before the next operation like add term to the filter or add route to the
> routing table is possible.
>

[Alia] Agreed and changed.

> High-throughput is always nice to have, but the quantification is a bit

[Alia] I know - it varies based on use-case and what's being controlled.

> config push? Or some other complex RPC?
> one and adding term is fastest create operation. IMO, NETCONF has no problem

[Alia] I know it depends on the back-end as well as the protocol.  Remember this
is the problem-statement and these are desired aspects.  I've been
trying to encourage
less wishy-washy numbers for use-cases, but without much success.
Should I just claim
a goal of a sustained rate of 10000 simple operations/sec?  OR bursts?

[Alia] I'll update it to:

"At a minimum, the I2RS Agent
          and associated router should be able to handle a
          considerable number of operations per second (for example
          10,000 per second to handle many individual subscriber
          routes changing simultaneously).
"
I'd like better wording and example though.

> - multiple interfaces per transaction 100/sec

[Alia] Hmm, I think this is too much detail for a problem-statement.  What about
for responsive:  "Within a sub-second time-scale, it should be
possible to complete simple operations (e.g. reading or writing a
single prefix)."

> but it is not spelled out.

[Alia] Can you clarify please?

> Dean
>
>
>
>
> _______________________________________________
> i2rs mailing list
> [email-masked]
> https://www.ietf.org/mailman/listinfo/i2rs
>

---

From: Alia Atlas <[email-masked]>
Date: Fri, 16 May 2014 17:21:28 -0400

Jamal,

I'd like to get some good numbers for use-cases to put in.

For operations, it's not clear to me that they will always be batched
in bulk as much as
is done for down to the forwarding plane - I think aspiring to
100,000s ops/sec is a bit
unlikely unless it is carefully crafted with good bulk operations.
I'm putting in 10,000 as
being more than we can generally do now with NetConf or CLI - but I'd
be happy to see
a use-case that justifies more.

Alia