Subject: [Dime] Comments about draft-fajardo-dime-interop-test-suite-00

From: "Tolga Asveren" <[email-masked]>
Date: Wed, 15 Mar 2006 14:14:20 -0500

I have the following comments about
draft-fajardo-dime-interop-test-suite-00:

1) There is no section about encoding/decoding of messages/AVPs. I believe
it could be useful to have a section just for that with positive and
negative test cases, e.g. messages/AVPs with invalid length, flag values,
padding etc. .

2) Things like "Reconnecting", are they really interoperability issues? They
seem to me totally local procedures. RFC3588 talks about the expected
behavior but would it break interoprability if somebody does not attempt a
reconnect at all?

3) The topology presented in 3.1.1.5. and the suggested of way of testing
seem overcomplex to me for failover/failback testing. Isn't something like
this enough:

              +------+
    +---------+Node B|
    |         +------+
 +--+---+
 |Node A|
 +--+---+
    |         +------+
    +---------+Node C|
              +------+

Node A sends messages to Node B. When it detects that Node B is down, it
retransmits messages to Node C with T-flag set.

4) 3.1.1.5
"o  Negative test to generate DIAMETER_UNABLE_TO_DELIVER on answer
      message from B to A when Destination-Host is set to C."
I didn't understand what we are trying to test here.

5) 3.1.1.5
For duplicate message detection -the last item in this section-
i- I wouldn't call it a negative test case, IMO it is a positive test case,
i.e. things are working the way they are supposed to work and nobody is
misbehaving.
ii- It seems like the suggesed way of testing won't allow the desired
behavior. Answers from NodeC will be still forwarded to NodeA. This would
both clean the pending request in the retransmission queue and even if no
watchdog is exchanged would help to keep the connection alive. I suggest the
following:
Configure asymmetric IP routing on NodeB after capability exchange is done,
i.e. it can receive packets from NodeA but can't send to NodeA. It can
send/receive to/from NodeC. Node A will send requests to NodeA till it
detects that NodeB is "down". All such requests will be forwarded to NodeC
and corresponding answers will be sent to NodeB, but they won't be forwarded
to NodeA. when NodeA failovers to NodeD, it will retransmit all requests it
sent before via NodeD to NodeC. They need to be detected as duplicates.

I think another test case for duplicate detection could be useful as well:
Put the interface on NodeB down, now it can't communciate neither with
NodeA, nor with NodeB. NodeA keeps sending till it detects the fialure.
NodeA detects failure, retransmits. NodeC shouldn't detect requests as
duplicate.

6) Why is "Peer State Machine" related tests -3.2.1- considered as optional.
I believe from a black-box behavior point of view, all nodes MUST comply
with peer state machine, i.e. in terms off expected sent/receive messages.

7) This is more a generic comment but it is easier to discuss about it in
the context of an example:
4.1.1.2.1
 o  Positive test for proper client session establishment.  Verify
      that sub-session id is supported and that the client can support
      event record generation at the least.  Verify that the client
      should at least be able to support DELIVER_AND_GRANT.  Test
      entities must be able to configure their server implementation to
      send this avp.  Must conform to Section 9.4 and 9.8.7 of
      [RFC3588].

I would think, we should have step-by-step instructions for multiple test
cases covering the behavior described in the RFC, for example what does it
mean "supporting Sub-Session Id"? There should be test case verifying this
which probably includes accounting messages in the same session for
different subsessions, where one can observe that state machine processing
is done properly for each of them.
This type of test case description is rpeated at a few more cases, I believe
it is hard to run a test case just based on that description, or in other
words, it doesn't tell much more than RFC3588 itself. this comment applies
largely for all Credit Control test cases as well. If we want to test local
application behavior as well, I would expect test cases with expected
message flows + local indications/observations. Something like -an example
for Credit Control-:

     Client               Server
   CCFH= CONTINUE            |
       |                     |
       |                     |
       |                     |
       | ----CCR------------>|
       |   (initial request) |
       |                     |
      Tx                     |
     Expires                 |
       |                     |
 <-Grant service             |
   to user                   |
       |                     |

8) Certain test cases are a bit too much involving into application logic.
An example:
4.2.1.8
 o  Positive test for a direct debiting enquiry with the CC Client
      including the requested units.  Verify that the CC Server just
      deducts the corresponding monetary amount from the end user's
      account without performing rating.  Verify that the granted
      service units can be of type time, volume, service specific, or
      money.
I don't think something like verifying that correct amount is deducted from
user account belongs to Diameter interoperability.

    Thanks,
    Tolga

_______________________________________________
DiME mailing list
[email-masked]
https://www1.ietf.org/mailman/listinfo/dime

---

From: "Tolga Asveren" <[email-masked]>
Date: Wed, 15 Mar 2006 15:10:56 -0500

Hi Victor,

[.. snip ..]
> ... or simply configure C to discard request from A.
[TOLGA]Yes, that would do too, but I personally think if we can run test
cases without code customization, it may be more handy.

---

From: Victor Fajardo <[email-masked]>
Date: Wed, 15 Mar 2006 15:26:09 -0500

>negative test cases, e.g. messages/AVPs with invalid length, flag values,
this could be an optional test case. though if this where to be 
included, i'm more interested in the test entity sending an answer with 
error bit set.

>behavior but would it break interoprability if somebody does not attempt a
i dont think a peer attempting reconnection will break interoperablity. 
this test can be move change to an optional test since its not mandated.

>seem overcomplex to me for failover/failback testing. Isn't something like
the topology is describe as such so that many vendors can participate 
during a single test scenario. this is in the interest of time.

This should be read:

"Negative test to generate DIAMETER_UNABLE_TO_DELIVER answer message from B to A when Destination-Host is set to C and link3 is disconnected"

victor

---

From: "Tolga Asveren" <[email-masked]>
Date: Wed, 15 Mar 2006 16:02:09 -0500

Oppssss, I realized that my comment wasn't right. Please see below.

> -----Original Message-----
> From: Tolga Asveren [mailto:[email-masked]]
> Sent: Wednesday, March 15, 2006 3:11 PM
> To: [email-masked]
> Subject: RE: [Dime] Comments about
> draft-fajardo-dime-interop-test-suite-00
>
>
> Hi Victor,
>
> [.. snip ..]
> > >5) 3.1.1.5
[TOLGA]Actually that won't do. If we drop requests from NodeA, NodeC won't
receive anything, so when requests are retransmitted, it will be the first
time they are received by NodeC and they shouldn't be detected as
duplicates -that IMO, should be another test case-. For this particular test
case, requests should arrive at NodeC but answers shouldn't be received by
NodeA.