Subject: [AAA-WG]: RE: Issue 235 - Enabling efficient accounting record uniqueness c hecking

From: "Patil Basavaraj (NET/Dallas)" <[email-masked]>
Date: Mon, 12 Nov 2001 17:41:44 -0600

Hi Pat,
 
Some clarifications: 
 
> Raj, 
> 
> just to be clear, I haven't rejected it yet. I marked it as a possible

> reject. Here's a question that lingers that really needs to be
> addressed before a decision can be made. 

Thanks, I'll try to see if I can provide more info regarding
this issue. Hopefully the explanation below will help in making some
decisions regarding whether to accept this as an issue.
 
> The problem that was identified by Jari is what does the server do if
it 
> receives the original Accounting message without the 'D' bit AFTER it
has 
> received, and processed, a retransmitted accounting message. It is
quite 
> clear in this case that this solution only handles one particular
problem, 
> but doesn't really help since in order to handle the scenario I just 
> mentioned, one needs to parse through all the records.
 
Well, as I mentioned in a "brief" form in the response to Jari:
 
 "Yes, this can happen. Since it has the D bit on, it would be sent
 towards the accounting record uniqueness checking operations, as a
 record whose duplicate would be searched among the other received
 accounting records within the (configurable) "accounting record
 uniqueness comparison time zone" (+-, i.e. the implementation should
 look a bit backward & forward on the time line).    Such accounting
 records with D bit set would likely be put to a separate buffer where
 the D records (a tiny minority) are waiting to be compared against
 all the others within (+-) the uniqueness comparison time limit."
 
Here's the corresponding procedure in *detail*.
Implementations could vary.
This is an adhoc written example description (with a step-by-step
process that shows for eg. how it can work), 
========================================================
 
1. The Diameter server receives the accounting record marked with the
   "D" bit. 
 
2. The server forwards it to the related accounting record checking
   entity that is responsible for checking accounting record
   uniqueness. 
 
3. There is not (yet) a duplicate accounting record that has been
   received and stored, in its for recent (eg. =within 24h)
   accounting record storage. (In this example 24 hours is the maximum
   client node down-time and the time limit configured for "uniqueness
   time zone window size" to the *PAST*.
 
4. The accounting record marked with the "D" bit is stored in a
   separate buffer that is dedicated to such records. It is maintained
   there as long as the "uniqueness time zone window size" has not
   expired for that record.
 
4. Subsequently the first accounting record sent (by client),
   non-D-marked (but duplicate!) pair of the above mentioned D-marked
   record is received by the Diameter server. There is now *no*
   obligation to "compare this non-D-flagged record aganist any other
   records"....
 
   BUT, there *is* a requirement to compare any D-marked accounting
   records in the "buffer dedicated for the recently received D-marked 
   records" against the accounting records received in a certain time
   limit  (within the configured 24h uniqueness time zone, like the
   just recently arrived non-D marked (but actually duplicate) record)
   Therefore: 
 
6. The D-marked accounting record is compared against the newly
   arrived non-D-marked duplicate record in the buffer; a match is
   found and the newly arrived record is deleted.
 
7. The D-marked record still continues to stay as a guardian in the
   "buffer dedicated for the recently received D-marked records",
   until the expiry of the timer for its validity (24h in this example).

   This record is not sent to the accounting record storage repository
   until the time "T" for duplicate checking has elapsed. If the
duplicate
   record has been sent to the repository, the "D" marked record is
   deleted at the end of time "T". 
 
   (Or, if the implementation is such that it is a rapid forwarder of
   acct records towards the accounting system repository, then just
   the copies of the D-marked accounting record headers with the
   related Session-Id AVP and Accounting-Record-Number AVP pairs are
   kept in the  "buffer for D-marked  records".)
 
The point here is that :
 
a) the non-D-marked records received by the Diameter server are *not*
   compared against all others but the D-marked records (waiting in
   the "buffer dedicated for the recvd D-marked records") *are*
   compared  against others (=future incoming records within the time
   window and the previously received records within the time window).
 
b) the D-marked records stay in that buffer the full length of the
   (max node downtime) time period, *even* if it is for eg. the first
   ever accounting record that arrived to that Diameter server and
   accounting record checking entity.
 
c) the comparison is done only within the +-(max node downtime) time
   window. 
 
d) as an implementation preference, the accounting record uniqueness
   checking entity could possibly immediately send the D-marked
   records to the accounting system repository, as long as copies of
   the headers and the Session-Id and Accounting-RecordNumber AVPs for
   their header stay as "guardians" in the "buffer dedicated for the
   recently received D-marked records" in the accounting record
   uniqueness checking entity. 
 
The general rational followed here:
The uniqueness checking system will perform uniqueness checks on only
the records (or their header + 2 AVP copies) that it has received ('D'
marked records). Hence there is no reason to check every record for
uniqueness but rather a very small subset of all accounting records.
This is where the significant gain in efficiency is achieved.
 
> I'd also claim that if hashed properly (per session id), the lookup is

> much smaller than the numbers you claim below. Also sending accounting

> records every 10s is something that is unheard of in the Internet
space. 

I mentioned the lengths of example AVP strings, but am not trying to
give an exact number for the average octet amount per AVP pair to be
compared since that is a very difficult thing to do unless there is a
specific hashing algorithm description on the table to discuss
about. I concur with you that within the checking of a Session-Id
string, there are several possibilities when trying to make that
comparison happen more efficiently than eg. a simple comparison that
looping through octet by octet offers.
But, I have tried to notify, that no matter what AVP octet string
comparison algorithm one uses, the same method can be used for *either*
of the 2 checking method alternatives.
Therefore the most important tangible numeric differentiator is that
the *amount of AVP pair comparisons in total*; and that is where the
big difference lies (~over a billion times less AVP pair comparisons
via alternative 1). 
And, that number is not related to the hashing and comparison
algorithms inside the AVP pair strings, because for uniqueness check,
one must compare all the target AVP pairs of set 1 to all the AVP
pairs of the comparison set 2, anyway. 
Finally, as I mentioned  in an earlier email, the structure of the
Session-Id record does not enable the most optimized hashing schemes.
 
In terms of trends emerging towards "hot" billing: 
At least some mobile network operators have voiced increasing interest
for more and more "hot" billing (for the ever increasing number of
prepaid customers and for accurate resource termination when the
prepaid account is exhausted). Also, there is a need to get "real
time" accurate accounting metering in place if possible, for packet
data networks. This is needed in order to be able to offer for
eg. Advice of Charge service with accurate numbers for subscribers. In
several countries, an operator can not offer AoC service, unless it is
relatively quite accurate (="true". 
 
For packet data this is not so straightforward as it is for circuit
switched communications, as the line cost is typically hevily dependent
on the exact volume transferred (and the packet data transfer can be
indeed very bursty in profile per a single subscriber terminal).
 
Furthermore, discussions have started in various accounting related
forums in relation to Application charging. Application level services
will in future be also possibly appearing in the subscribers bill :( 
The application layer services may have a premium price, meaning that
the amount of network service costs during a network connection for a
subscriber may vary in a more bursty way than has been the case so far.
 
To encourage the end users to use such services, the operator needs to
offer AoC type service so that the network user can see his balance in
almost real time. Only then will the end user be willing to use such
services. As a basic enabler for that, the operator needs an
accounting protocol that does not have difficulty w.r.t frequent
accounting record transmissions. It would be good for Diameter to be
equally capable in supporting rapid transmission of acct. records as
it is for infrequent accounting messaging.
 
Of course, if some carriers offer just a flat fee, (e.g. XX dollars
per month) then an exact (near real-time) account metering would not
be a requirement at all. (But then, even the accounting metering for
that subscriber would not be needed at all.)
 
> Perhaps carriers are interested in this, but I've never heard anyone 
> make such a claim. I would even argue that a high density box would 
> probably even need a separate Accounting processor :) 
 
It is indeed so that for the packet data the Accounting system may
easily be the bottle neck. I believe that for several operators the
billing system is among the most critical parts in the network service
capacity chain. 
 
> So if we can get through the above issue, perhaps I would feel more 
> comfortable with this issue (or at least the proposed resolution)
since 
> I'm not convinced yet there is an issue. 
> 
> PatC 
 
The most relevant point w.r.t Issue 235 is:
-  Accounting data processing performance.
By requiring to mandatorily  check *just* the potentially duplicated
accounting packets, there is a tremendous performance gain over the
other alternative where every AVP pair is subjected to equal suspicion
when it comes to the uniqueness checking.  The performance gain is
extremely high. 
 
Furthermore, the alternative 1 solution (to issue 235) is a very small
and simple enhancement. The mechanism proposed  would still *allow*
the alternative 2 way of full "robotic" checking for *every* AVP pair
against *every other* AVP pair, if required for any reason.
This uniqueness checking is not actually even a core protocol feature
here, but an enabler, to allow the Diameter protocol user/system to
get as much performance into beneficial net use as is available, at
least in this checking task where the performance gain seems very big.
 
Thanks for your questions. I hope the responses clarify the intent in
the proposal.
 
-Basavaraj