Subject: [Dime] trying to understand auditing problem

From: "Tolga Asveren" <[email-masked]>
Date: Thu, 14 Dec 2006 13:52:18 -0500

Here are some questions I came up with, while I was trying to understand the
concept of auditing as described in draft-bodin-dime-auditing-reqs-01.txt:

1) I think the goal is not (and IMO should not be) to prevent session
leaking. For example, if server or client is down, IMO application layer
timers on the other side are supposed to take care of lack of
requests/responses and delete session accordingly. The application layer
protocol needs to be designed properly to take care of this type of
situation.

2) Is the goal to allow restarting nodes to gather enough information to
process subsequent incoming requests? I would think carrying relevant
information in the requests to allow them to be processed by backup
instances could be a more handy approach, i.e. basically allowing servers to
be stateless (at least from protocol processing point of view).

3)  AFAICS, the third issue is, how does a backup client knows, which
sessions were alive, what data was associated with them etc..., so that it
can continue to manage the service provided to the customer. I know the
exact approach could be dependent on the service we are talking about but I
will use DCCA/prepaid call as an example:

 +---+    +--------+
 |   |    |        |
 | S |::::|DCCA    |          +--------+
 | e |    |Client-1+----------+        |
 | r |    +--------+          |        |
 | v |                        | DCCA   |
 | i |                        | Server |
 | c |    +--------+          |        |
 | e |    |        +----------+        |
 |   |::::|DCCA    |          +--------+
 | L |    |Client-2|
 | o |    +--------+
 | g |
 | i |
 | c |
 +---+

Let's say there are 3 calls, which are using prepaid service. Credit control
for Call-1 and Call-2 are handled by DCCA Client-1 and for Call-3 by DCCA
Client-2. Now, Client1-fails, Client-2 needs to handle Call-1 and Call-2 as
well.

The first thing Client-2 needs to do is to determine which calls were
handled by Client-1. I guess, this can be done in a straightforward way via
communication between Client-2 and Service Logic component (assuming that
this data is not synchronized between Client-1 and Client-2). Now, Client-2
can start a new session for these calls and perform credit control
operation. The sessions on DCCA server, which were opened by requests from
client-1 will be cleaned up after Tcc expires.

Overall, it seems to me that with careful protocol design, it is possible to
addres 3) as well without a need of a special mechanism.

    Thanks,
    Tolga

_______________________________________________
DiME mailing list
[email-masked]
https://www1.ietf.org/mailman/listinfo/dime

---

From: "Ulf Bodin" <[email-masked]>
Date: Thu, 14 Dec 2006 21:41:36 +0100

Hi Tolga, 

Pls find my replies below. 

Best regards, 
Ulf 

-----Original Message-----
From: Tolga Asveren [mailto:[email-masked]] 
Sent: den 14 december 2006 19:52
To: [email-masked]
Subject: [Dime] trying to understand auditing problem

Here are some questions I came up with, while I was trying to understand
the concept of auditing as described in
draft-bodin-dime-auditing-reqs-01.txt:

Ulf: I'm not sure how this can be handled in case of hard-state. I.e.
without any messages updating timers for sessions remaining
valid/active. In case long timers would be used to approach a hard-state
behaviour this would also increase the period of inconsistency (which
depending on the application can be costy and even incopatible with the
service offered). 

2) Is the goal to allow restarting nodes to gather enough information to
process subsequent incoming requests? I would think carrying relevant
information in the requests to allow them to be processed by backup
instances could be a more handy approach, i.e. basically allowing
servers to be stateless (at least from protocol processing point of
view).

Ulf: A failover to a cold standby basically corresponds to having the
server stateless. For this particular scenario I think we just need a
mechanism triggering clients to actuall resubmit their previously issued
requests with all relevant information. These resubmitted requests must
however be separated from new requests. I.e. since the server wont
recognize any session IDs it must be given another indication of which
sessions are new and which ones that a just re-established. 

3)  AFAICS, the third issue is, how does a backup client knows, which
sessions were alive, what data was associated with them etc..., so that
it can continue to manage the service provided to the customer. I know
the exact approach could be dependent on the service we are talking
about but I will use DCCA/prepaid call as an example:

 +---+    +--------+
 |   |    |        |

Let's say there are 3 calls, which are using prepaid service. Credit
control for Call-1 and Call-2 are handled by DCCA Client-1 and for
Call-3 by DCCA Client-2. Now, Client1-fails, Client-2 needs to handle
Call-1 and Call-2 as well.

The first thing Client-2 needs to do is to determine which calls were
handled by Client-1. I guess, this can be done in a straightforward way
via communication between Client-2 and Service Logic component (assuming
that this data is not synchronized between Client-1 and Client-2). Now,
Client-2 can start a new session for these calls and perform credit
control operation. The sessions on DCCA server, which were opened by
requests from
client-1 will be cleaned up after Tcc expires.

Ulf: If I understand this scenario correctly I believe it's similar to
the second one. I.e. a trigger from Client 2 to the service logic making
it re-submit requests for active sessions and tag them as re-establish
requests rather than new requests would probably be enough. 

Overall, it seems to me that with careful protocol design, it is
possible to addres 3) as well without a need of a special mechanism.

    Thanks,
    Tolga

---

From: "Tolga Asveren" <[email-masked]>
Date: Thu, 14 Dec 2006 16:28:15 -0500

Hi Ulf,

Please see below for some more questions/comments.

  Thanks,
  Tolga

> From: Ulf Bodin [mailto:[email-masked]]
> Sent: Thursday, December 14, 2006 3:42 PM
> To: [email-masked]
> Subject: RE: [Dime] trying to understand auditing problem
>
>
> Hi Tolga,
>
[TOLGA] O.K., now it is more clear to me. AFAICS, using timers translates to
a soft-state scenario.

For example for DCCA :
a)For event based scenario, there is no issue, it is just a single
transaction.
b)For session based scenario, client needs periodically to send new
requests. This maximum value for this period is communciated from server and
server is running Tcc timer. If Tcc expires and no new request is received
yet, the session is freed. Similarly client uses Tx timer while waiting for
replies.

Overall, hard-state approach seems a risky strategy to me, for example what
happens if client dies and there is no backup server? Will the sessions on
the server stay there forever? IMHO, use of timers at both client and server
side makes life really easy.
> sessions are new and which ones that a just re-established.
[TOLGA]If the clients are sending new requests periodically and if the
requests contain enough information for proper processing even if they are
sent to a cold standby this probably should work. If the sessions for the
reqeusts are not known by the cold standby instance yet, this shouldn't be a
problem. It just will create a new session and perform processing according
to the contents of the received request. It seems this item is again closely
related whether a timer is running on the client side.
> about but I will use DCCA/prepaid call as an example:
[TOLGA]On this item, I tried to evaluate the situation from client point of
view (you are right, this is the same scenario as 2, this time from client
perspective). Basically how Client-2 determines calls which were served by
Client-1 is really implementation dependent. It definetly can be done
though, the calls (or any resource in that sense) is controlled by some
service logic, which has the information about what is currently provided to
the users. What Client-2 needs to do is just to create new sessions for
these service instances. "Old" sessions on the DCCA server will expire upon
expiry of Tcc.

---

From: "Ulf Bodin" <[email-masked]>
Date: Fri, 15 Dec 2006 18:31:06 +0100

Hi Tolga, 

Further replies below. 

Best regards, 
Ulf 

>
> Here are some questions I came up with, while I was trying to 
> understand the concept of auditing as described in
> leaking. For example, if server or client is down, IMO application 
> layer timers on the other side are supposed to take care of lack of 
> requests/responses and delete session accordingly. The application 
> layer protocol needs to be designed properly to take care of this type

> of situation.
> valid/active. In case long timers would be used to approach a 
> hard-state behaviour this would also increase the period of 
> inconsistency (which depending on the application can be costy and 
> even incopatible with the service offered).
[TOLGA] O.K., now it is more clear to me. AFAICS, using timers
translates to a soft-state scenario.

For example for DCCA :
a)For event based scenario, there is no issue, it is just a single
transaction.
b)For session based scenario, client needs periodically to send new
requests. This maximum value for this period is communciated from server
and server is running Tcc timer. If Tcc expires and no new request is
received yet, the session is freed. Similarly client uses Tx timer while
waiting for replies.

Overall, hard-state approach seems a risky strategy to me, for example
what happens if client dies and there is no backup server? Will the
sessions on the server stay there forever? IMHO, use of timers at both
client and server side makes life really easy.

Ulf2: I assume you refer to a dead client without any backup client.
Then, if a client dies the server certainly needs a strategy for how to
handle hard states established by a confirmed dead client (i.e. if there
is no backup client these states need to be terminated). If instead
there is a backup client taking over a (short) while, it should be able
to ensure that it's in sync with the server. 
>
> 2) Is the goal to allow restarting nodes to gather enough information 
> to process subsequent incoming requests? I would think carrying 
> relevant information in the requests to allow them to be processed by 
> backup instances could be a more handy approach, i.e. basically 
> allowing servers to be stateless (at least from protocol processing 
> point of view).
>
> mechanism triggering clients to actuall resubmit their previously 
> issued requests with all relevant information. These resubmitted 
> requests must however be separated from new requests. I.e. since the 
> server wont recognize any session IDs it must be given another 
> indication of which sessions are new and which ones that a just
re-established.
[TOLGA]If the clients are sending new requests periodically and if the
requests contain enough information for proper processing even if they
are sent to a cold standby this probably should work. If the sessions
for the reqeusts are not known by the cold standby instance yet, this
shouldn't be a problem. It just will create a new session and perform
processing according to the contents of the received request. It seems
this item is again closely related whether a timer is running on the
client side.

Ulf2: Yes, with the addition that a trigger to resubmit requests is
needed and an indication of that the resubmitted requests are for lost
session states and not for new sessions (i.e. the server may need to
differentiate between these two types of session requests to provide its
service properly). 
> sessions were alive, what data was associated with them etc..., so 
> that it can continue to manage the service provided to the customer. I

> know the exact approach could be dependent on the service we are 
> talking about but I will use DCCA/prepaid call as an example:
>
>  +---+    +--------+
>  |   |    |        |
> handled by Client-1. I guess, this can be done in a straightforward 
> way via communication between Client-2 and Service Logic component 
> (assuming that this data is not synchronized between Client-1 and 
> Client-2). Now,

> the second one. I.e. a trigger from Client 2 to the service logic 
> making it re-submit requests for active sessions and tag them as 
> re-establish requests rather than new requests would probably be
enough.
[TOLGA]On this item, I tried to evaluate the situation from client point
of view (you are right, this is the same scenario as 2, this time from
client perspective). Basically how Client-2 determines calls which were
served by
Client-1 is really implementation dependent. It definetly can be done
though, the calls (or any resource in that sense) is controlled by some
service logic, which has the information about what is currently
provided to the users. What Client-2 needs to do is just to create new
sessions for these service instances. "Old" sessions on the DCCA server
will expire upon expiry of Tcc.
Ulf2: Yes, I agree. For this scenario the service logic provides the
trigger to Client 2 and since the server hasn't done a failover it
should recognize the session ID of Call 1 and Call 2 as existing ones.
If instead the server would have done a failover such indication would
IMHO be needed.

---

From: "Tolga Asveren" <[email-masked]>
Date: Fri, 15 Dec 2006 15:03:39 -0500

Hi Ulf,

Please see below for more comments/questions.

  Thanks,
  Tolga

[..snip..]
> to ensure that it's in sync with the server.
[TOLGA]Yes, sorry for the typo, the scenario was a dead client with no
backup client. I guess  confirming (or better said believing with a very
high probability) that the client is dead will require a timer on server
side anyhow, transforming the scenario to a soft-state case.

[..snip..]
> differentiate between these two types of session requests to provide its
[TOLGA]I guess with careful protocol design one can avoid the need to
indicate that resubmitted requests are for lost session states. For example
in DCCA, if a client dies, a backup instance takes over and starts new
sessions for the service instances it took over. I guess no harm would be
done even for session based scenario, except service provider not being able
to charge for the last service period at a maximum, which really shouldn't
be a big amount. Old sessions in the server just will expire upon timer
expiry.

> If instead the server would have done a failover such indication would
[TOLGA]Will there be any cases where Service Logic won't be controlling the
service resources? I guess the answer is negative because otherwise Service
Logic wouldn't be able to  release resources allocated for a specific user
upon a negative answer from DCCA server. So, I assume Service Logic can
always provide a trigger for a backup client taking over responsibilites of
a failed one.

I think it is possible that the server does not need to know anything about
whether the sessions are a replacement for existing sessions. Old ones just
will expire if a timer is utilized. Again it depends on the design of the
application layer protocol. Actually, in general IMHO proper application
layer protocol design is important to deal with issues audit mechanism talks
about and could provide a satisfactory solution for most of the cases.

---

From: "Ulf Bodin" <[email-masked]>
Date: Mon, 18 Dec 2006 14:50:42 +0100

Hi Tolga, 

Clearly it's quite easy to find examples in which Diameter clients and
their server can rely on soft-state to ensure that they after timer
expiration return to the same view on the set of sessions being active.
In my view soft-state without auditing is preferable over soft-state
with auditing for synchronization or hard-state in many cases for
several reasons. 

For example, for high-volume and short holding-time
applications/sessions it would be troublesome to audit high volumes of
states while serving new session requests in parallel. That is, auditing
will consume resources and take time. Consequently, many audited session
states are likely to have been terminated by the application before
being audited. Also, for applications that can accept periods of
client/server inconstency, soft-state without auditing for
synchronization is most likely the way to go. 

On the other hand, lower-volume and longer-holding time
applications/sessions auditing can allow for more precise services
offered by Dimeter server systems. Also, to some extent, auditing can
relax requirements on state replication at both clients and server. An
example of such a scenario is when an instance of RACS (Resource and
Admisison Control Subsystem as defined by ETSI TISPAN) is to handle
resource reservations for leased-line services. 

For simplicitly, say that only two leased-line service nodes and two
RACS nodes exist, one active and one backup node of each kind. The
active leased-line service Diameter client node establishes a session
with the active RACS Diameter server node. This corresponding session
state in RACS must not be lost for any reason (e.g. the contractual
agreement for the leased-line service makes it very costly to lose
resource reservation states for this service). Hence, to protect against
state loss or at least periods of state inconsistency, replication is
required at the client side, at the server side, or both. Through
auditing, states need not be replicated at both client and server and/or
replication can be asyncronous. 

In case of hard-state, I agree that states must be terminated in case
the backup instance can't be activated. In my view this fact does
however not transform the hard-state to a soft-state. For example,
failing to activate a hot-standby backup client or server should indeed
result in states being terminated either explicitly by a management
operation or automatically through a timeout. However, in case the
backup is activated successfully, auditing is useful to ensure
synchronisation without waiting for timer expiration. 

The problem of state synchronization may be possible to engineer around,
but it's my firm belief that that an auditing capability in Diameter
would be useful and allow for more attractive overall solutions compared
to such engineering approaches. Again, I don't believe auditing to the
best approach for all applications, but rather for particular
applications with specific needs (e.g. such as RACS resource
reservations for leased-lines). 

Best regards, 
Ulf 

-----Original Message-----
From: Tolga Asveren [mailto:[email-masked]] 
Sent: den 15 december 2006 21:04
To: [email-masked]
Subject: RE: [Dime] trying to understand auditing problem

Hi Ulf,

  Thanks,
  Tolga

[..snip..]

> Then, if a client dies the server certainly needs a strategy for how 
> to handle hard states established by a confirmed dead client (i.e. if 
> there is no backup client these states need to be terminated). If 
> instead there is a backup client taking over a (short) while, it 
> should be able to ensure that it's in sync with the server.

[..snip..]

> differentiate between these two types of session requests to provide 
> its service properly).
[TOLGA]I guess with careful protocol design one can avoid the need to
indicate that resubmitted requests are for lost session states. For
example in DCCA, if a client dies, a backup instance takes over and
starts new sessions for the service instances it took over. I guess no
harm would be done even for session based scenario, except service
provider not being able to charge for the last service period at a
maximum, which really shouldn't be a big amount. Old sessions in the
server just will expire upon timer expiry.

> >
> > 3)  AFAICS, the third issue is, how does a backup client knows, 
> > which sessions were alive, what data was associated with them 
> > etc..., so that it can continue to manage the service provided to 
> > the customer. I
>

> > The first thing Client-2 needs to do is to determine which calls 
> > were handled by Client-1. I guess, this can be done in a 
> > straightforward way via communication between Client-2 and Service 
> > Logic component (assuming that this data is not synchronized between

> > Client-1 and Client-2). Now,

> [TOLGA]On this item, I tried to evaluate the situation from client 
> point of view (you are right, this is the same scenario as 2, this 
> time from client perspective). Basically how Client-2 determines calls

> which were served by
> though, the calls (or any resource in that sense) is controlled by 
> some service logic, which has the information about what is currently 

> sessions for these service instances. "Old" sessions on the DCCA 
> server will expire upon expiry of Tcc.

> IMHO be needed.
[TOLGA]Will there be any cases where Service Logic won't be controlling
the service resources? I guess the answer is negative because otherwise
Service Logic wouldn't be able to  release resources allocated for a
specific user upon a negative answer from DCCA server. So, I assume
Service Logic can always provide a trigger for a backup client taking
over responsibilites of a failed one.

I think it is possible that the server does not need to know anything
about whether the sessions are a replacement for existing sessions. Old
ones just will expire if a timer is utilized. Again it depends on the
design of the application layer protocol. Actually, in general IMHO
proper application layer protocol design is important to deal with
issues audit mechanism talks about and could provide a satisfactory
solution for most of the cases.

---

From: "Tolga Asveren" <[email-masked]>
Date: Mon, 18 Dec 2006 17:35:28 -0500

Hi Ulf,

I agree with you on most of the issues with slight disagreement on
applicability of certain mechanisms for different types of applications.
IMHO
a) The effect of application level timers and the role they play to prevent
session leaking/failure handling should be documented. I think "Diameter
Application Design Guidelines Document" which is an item in the charter is a
good place for it. Basically what has been discussed in this thread would be
useful material for anybody designing a new Diameter application.
b) The audit document may provide a bit more information about other
possible mechanisms to deal with the problems it tries to address. A few
short paragraphs and a reference to the relevant section in the Application
Design Guidelines document could be a good idea.

    Thanks,
    Tolga

> Sent: Monday, December 18, 2006 8:51 AM
> To: [email-masked]

---

From: "Ulf Bodin" <[email-masked]>
Date: Tue, 19 Dec 2006 11:02:55 +0100

Hi Tolga, 

I completely agree to documenting different alternatives to prevent
session leaking and inconsistency related to resilience and failovers.
My first understanding (from quickly reading the charter) was that the
"Diameter Application Design Guidelines Document" is as defined in the
charter not targeted to cover these kind of issues (more or less
implicitly said though), but I do believe it should. 

I also agree to having a few paragraphs on alternative approaches to
auditing in the auditing requirements document with reference to the
application design guide. Eventually, assuming we do a specification on
Diameter auditing, the reference to the application design guide should
be brought into that specification. 

The approach I believe we should now follow is to make an .03 of the
auditing requirements draft and a first version of an auditing solutions
document (draft) based on the expired
draft-calhoun-diameter-res-mgmt-08.txt. I would also be interested to be
involved in putting together the application design guide document. 

QUESTION (maybe to the chairs): what are the current plans to progress
this applications design document for Diameter (i.e. beyond what's said
in the charter).

Best regards, 
Ulf  

-----Original Message-----
From: Tolga Asveren [mailto:[email-masked]] 
Sent: den 18 december 2006 23:35
To: [email-masked]
Subject: RE: [Dime] trying to understand auditing problem

Hi Ulf,

I agree with you on most of the issues with slight disagreement on
applicability of certain mechanisms for different types of applications.
IMHO
a) The effect of application level timers and the role they play to
prevent session leaking/failure handling should be documented. I think
"Diameter Application Design Guidelines Document" which is an item in
the charter is a good place for it. Basically what has been discussed in
this thread would be useful material for anybody designing a new
Diameter application.
b) The audit document may provide a bit more information about other
possible mechanisms to deal with the problems it tries to address. A few
short paragraphs and a reference to the relevant section in the
Application Design Guidelines document could be a good idea.

    Thanks,
    Tolga

> states while serving new session requests in parallel. That is, 
> auditing will consume resources and take time. Consequently, many 
> audited session states are likely to have been terminated by the 
> application before being audited. Also, for applications that can 
> accept periods of client/server inconstency, soft-state without 
> auditing for synchronization is most likely the way to go.
>

> resource reservation states for this service). Hence, to protect 
> against state loss or at least periods of state inconsistency, 
> replication is required at the client side, at the server side, or 
> both. Through auditing, states need not be replicated at both client 
> and server and/or replication can be asyncronous.
>
> failing to activate a hot-standby backup client or server should 
> indeed result in states being terminated either explicitly by a 
> management operation or automatically through a timeout. However, in 
> case the backup is activated successfully, auditing is useful to 
> ensure synchronisation without waiting for timer expiration.
>
> The problem of state synchronization may be possible to engineer 
> around, but it's my firm belief that that an auditing capability in 
> Diameter would be useful and allow for more attractive overall 
> solutions compared to such engineering approaches. Again, I don't 
> believe auditing to the best approach for all applications, but rather

> for particular applications with specific needs (e.g. such as RACS 
> resource reservations for leased-lines).
>
> Best regards,
> Ulf
>
>
>

> > to handle hard states established by a confirmed dead client (i.e. 
> > if there is no backup client these states need to be terminated). If

> backup client. I guess  confirming (or better said believing with a 
> very high probability) that the client is dead will require a timer on

> server side anyhow, transforming the scenario to a soft-state case.
>
>
>
> [..snip..]
> > are sent to a cold standby this probably should work. If the 
> > sessions for the reqeusts are not known by the cold standby instance

> > yet, this shouldn't be a problem. It just will create a new session 
> > and perform processing according to the contents of the received 
> > request. It seems
>

> > > Client-2 can start a new session for these calls and perform 
> > > credit control operation. The sessions on DCCA server, which were 
> > > opened by
>
> > > requests from
> > Client-1 is really implementation dependent. It definetly can be 
> > done though, the calls (or any resource in that sense) is controlled

> > by some service logic, which has the information about what is 
> > currently provided to the users. What Client-2 needs to do is just 
> > to create new
>

> [TOLGA]Will there be any cases where Service Logic won't be 
> controlling the service resources? I guess the answer is negative 
> because otherwise Service Logic wouldn't be able to  release resources

> allocated for a specific user upon a negative answer from DCCA server.

> So, I assume Service Logic can always provide a trigger for a backup 
> client taking over responsibilites of a failed one.
>
> about whether the sessions are a replacement for existing sessions. 
> Old ones just will expire if a timer is utilized. Again it depends on 
> the design of the application layer protocol. Actually, in general 
> IMHO proper application layer protocol design is important to deal 
> with issues audit mechanism talks about and could provide a 
> satisfactory solution for most of the cases.
>
>