Subject: RE: WG Last Call on the LDAPv3 Directory Replication I-D

From: "Paul Leach" <[email-masked]>
Date: Tue, 20 Feb 2001 09:35:44 -0800

Very roughly, if there are N replicas, and each can tolerate an update
rate (if it were the only replica) of R, then the total update rate
supportable by the replicated directory would be R/N.

I do not believe that the replication frequency has much to do with it,
unless a low-ish frequency causes many updates to be transferred per
replication interval and thereby leads to increased efficiency (due to
some batching effect or other).

> -----Original Message-----
> From: John Strassner [mailto:[email-masked]]
> Sent: Tuesday, February 20, 2001 8:21 AM
> To: Ed Reed
> Cc: [email-masked]; [email-masked];
> [email-masked]
> Subject: RE: WG Last Call on the LDAPv3 Directory Replication I-D
> 
> 
> This seems reasonable, but I would suggest that we quantify 
> this, perhaps 
> in terms relative to the (average) replication frequency of 
> the system. 
> Thoughts?
> 
> regards,
> John
> 
> At 08:41 AM 2/20/2001 -0700, Ed Reed wrote:
> >Hmmm...I've found a point of agreement with Albert ;-)... see [eer]
> >
> >=================
> >Ed Reed
> >Reed-Matthews, Inc.
> >+1 801 796 7065
> >http://www.Reed-Matthews.COM
> >
> > >>> "Albert Langer" <[email-masked]> 02/17/01 
> > 01:30PM >>>
> >[Kurt]
> >At 04:39 PM 2/17/01 +1100, Albert Langer wrote:
> > >Put succinctly: LDUP directories won't interoperate. Period.
> >
> >I would say that two LDUP peers could interoperate (from
> >a protocol perspective) but that this, in itself, does
> >not:
> >         1) preserve LDAP/X.500 semantics nor
> >         2) provide "equally capable" replicas.
> >
> >The former issue can not be resolved unless one requires
> >transactional consistency in multiple master replication.
> >The latter issue can be tackled by writing a very tight
> >applicability statements.
> >
> >[Albert]
> >
> >...snip...
> >
> >A necessary consequence of multi-master, as opposed to single master,
> >is that in addition to transient inconsitencies between data 
> read from
> >different directory servers there will also be problems with 
> data read
> >from a single server. There are 2 options possible for multi-master:
> >
> >1) Conflicting updates to the same entry are prioritized so that only
> >one succeeds and the others are rolled back. That necessarily results
> >in different behaviour to single master directories but is not
> >fundamentally inconsistent with LDAP semantics and does not prevent
> >interoperability. Problems arising can be resolved by users 
> rather than
> >sysadmins by notifying them of lost updates (whether due to 
> name conflicts
> >or
> >other conflicts). An applicability statement should clearly indicate
> >that multi-master is unsuitable in any situation with a high rate of
> >concurrent changes to the same entries.
> >
> >[eer] I certainly agree that we should make clear (if we have not
> >already), that multi-master is unsuitable in any situation with a
> >high rate (say, frequently faster than the time it takes for a change
> >to fully propagate to all registered replicas) of concurrent changes
> >to the same entries at different master replicas.
> >
> >The "frequently..." qualifier represents my own bias as to what
> >is "reasonable" and might be changed to simply say "faster than...".
> >
> >[/eer]
>

---

From: Jerry Combs <[email-masked]>
Date: Tue, 20 Feb 2001 14:09:38 -0800

This assumes that replication is implemented as a 1 to N function. If
transitive replication was implemented instead, each replicas efficiency
could be maintained by limiting the number of replicas that it communicates
to directly. While at low update rates transitive synchronization lengthens
the update interval, at high update rates it is vastly more efficient. Lack
of transitive synchronization will limit LDUP scalability. Transitive synch
also allows you to minimize the impact of slow communication links.

JC
Principal Consultant
[email-masked]

---

From: "Albert Langer" <[email-masked]>
Date: Tue, 27 Feb 2001 14:37:28 +1100

[Paul]

[Albert]
I don't understand that and am worried that as I have no implementation
experience and each of Paul and Ed are or have been involved with a major
vendor with a large deployment of multi-master servers, I may have
missed something fundamental about deployment issues.

The original below was about unsuitably high rates of concurrent
changes to the same entry. Paul's formula S ~ R/N indicates
that the "very rough" approximation of the supportable total update rate
S, decreases as the number of replicas increases. That is consistent
with my understanding that the more opportunities for concurrency
there are, the more problems. If there was only 1 replica, there
would be no problems arising from conflicts. If every DUA was a DSA,
there would be a lot.

But for exactly the same reason I would agree with the implication
of Ed's comment that S decreases as the diameter of the replication
network increases - ie that replication frequency has just as much
to do with it, contrary to Paul's statement that frequency has
little to do with it.

Paul's response seems to be sliding from estimating S in the light
of conflict problems in the first paragraph to the quite separate
question of estimating what total update rate, R, can be handled
at all in the second.

On that issue I would have thought that "very roughly" R is
independent of N, with no gain and only a small loss from
increasing N by adding more replicas.

My assumption is that R is primarily limited by whichever of
the various constraints such as disk seek times, CPU
processing (mainly crypto), and network bandwidth
is actually a "tight" or "binding" constraint rather
than being slack.

Of these, there has to be enough bandwidth available for
the peak traffic to at least trickle through in the offpeak
periods. Therefore any constraint on bandwidth would primarily
affect the replication diameter by pushing more traffic into
offpeak periods in order to carry it at all, thereby
delaying it.

Likewise limitations arising from the number of disk spindles
available would primarily have the same effect - forcing you
to spread the traffic over a broader peak by splitting the
available disk spindles over a larger number of replicas,
at the cost of delaying it and thus increasing conflict
problems. In that context load sharing strategies just to handle
the write volume don't make much sense to me given that you still
have to write the update to disk on every replica. You would
need to be faced with peaks so sharply spiked and rates so
intense that the cost of writing *both* to a logging spindle
that is not limited by disk seek times *and* to the random
access spindles that are, for *every* replica on *every*
update, is offset by the gains from spreading out those
peaks a bit by having to handle only R/N of the traffic
immediately at each replica and being able to delay
(but not avoid) handling the remainder by pushing it
outside the peak spikes via logging spindles and
replication delays. Those would have to be very sharp
spikes indeed!

Thus I am puzzled by Ed's earlier anecdote about load
sharing of updates. Why would anyone want to do that?
To me one needs multi-master for local availability
and cheaper failover, not for load sharing on updates.

The remaining possibly binding constraint, from CPU
crypto processing, seems to me highly dependendent
on batching effects due to startup costs initiating
a secure association for each replication session
and the overhead of update vectors for each such
session.

Any comments?

PS I certainly agree with Rick that no attempt to
quantify any of this should be made in the
requirements draft.