Subject: Re: Proposed requirements not met by current architecture and URP proposals - part 2 (end)

From: <[email-masked]>
Date: Sun, 25 Feb 2001 18:41:07 -0500

What's going on here?  I read some of this to be
saying "the requirements are wrong" which was not
supposed to be the scope of this discussion.  Now
maybe I'm just misreading, so clarification would
help...

Again <rm>...</rm> and [snip] used to shorten list

---- Original message ----
>Date: Mon, 26 Feb 2001 08:46:07 +1100
>
>Continuing from part 1, which defines the categories used:
>
>http://www.imc.org/ietf-ldup/mail-archive/msg00891.html
>
>* B) Fixable Show Stopper *
>
>3) Eventual Convergence
>
>G1.  LDAP Replication MUST support models 2 (Eventual 
Consistency) and
>3 (Limited Effort Eventual Consistency) above.
>
>M3.  A value shared between replicas in a replica ring must 
eventually
>converge to the same value on all of its constituent 
replicas.
>
>MM6.  Multi-master replication MUST support convergence of 
the values
>of attributes and entries.  Convergence may result in an 
event as
>described in MM5.
>
>Each of these requires "eventual convergence" (the wording 
and redundancy
>could be improved - as is true of other items, without 
further comment
>below).

<rm>I disagree.  The requirements are specific to the part of
the discussion taking place.

[snip]</rm>

>6) System and Network Performance
>
>G3.  LDAP replication SHOULD have minimal impact on both the 
system and
>network performance.
>
>MM1.  The replication protocol SHOULD NOT saturate the 
network with
>redundant or unnecessary entry replication.

<rm>[snip...] but I'm keeping the above to refer to later</rm>

>7) Missing
>
>P1.  The replication protocol MUST provide for recovery and
>rescheduling of a replication session due to replication 
initiation
>conflicts (e.g. consumer busy replicating with other 
servers) and or
>loss of connection (e.g. supplier cannot reach a replica).
>
>P3.  The LDAP replication protocol MUST allow for full 
update to
>facilitate replica initialization and reset loading 
utilizing a
>standardized format such as LDIF [RFC2849] format.
>
>AM2.  Each copy of a replica MUST maintain audit history 
information of
>which servers it has replicated with and which servers have 
replicated
>with it.
>
>AM5. A mechanism to fix differences between replicas without 
triggering
>new replica cycles SHOULD be provided.
>
>No mechanisms appear to have been specified. It is unclear 
what
>document they should be specified in, but they all involve 
some interaction
>with URP. 

<rm>The requirement document should not mandate
implementation mechanisms.

[snip]...</rm>

>11) Design constraint on orderinig
>
>MM7.  Multi-master conflict resolution MUST NOT depend on 
the in-order
>arrival of changes at a replica to assure eventual 
convergence.
>
>SM2.  The master replica in a Single Master system SHOULD 
send all
>changes to read-only replicas in the order in which the 
master applied
>them.
>
>(Appendix B.2 explains that requirement SM2 is intentionally 
not a
>MUST and does not apply to multi-master at all)
>
>URP does not meet SM2 for Single Master but does meet MM7, 
at the expense
>of conflicting requirements G3 and MM1. Appendix B.2 does 
justify SM2,
>although the justification points to MUST and the 
explanation does
>not justify not adopting the same for multi-master and
>simply dropping MM7.
>
>Decisions about ordering are a design issue, not 
requirements.
>Design constraints should not be included in requirements 
without
>a clear reference to corresponding actual requirements.

<rm>Uh, you answered this yourself, didn't you? Decisions
about ordering are not a design issue in the single master
case, but are a design issue in the multi-master case.
Therefore, order can be required for single master but
not for the multi-master case</rm>
>
>12) Design constraint on update tracking
>
>P2.  The supplier MUST track updates sent to the consumer 
and not
>resend already acknowledged ones, even in the event of 
recovery from a
>failed replica cycle.
>
>This is another unecessary design constraint. It is not 
satisfied by
>URP and there does not appear to be anything unsound about 
the URP approach
>of
>having the consumer track updates received rather than the 
supplier
>tracking updates sent.

<rm>No, this breaks G3 and MM1 above</rm>

>13) Transactional Consistency
>
>G2.  LDAP Replication SHOULD NOT preclude support for model 1
>(Transactional Consistency) in the future.
>
>P7.  The protocol SHOULD NOT preclude support of 
Transactional
>Consistency (model 1).
>
>Model 1: Transactional Consistency -- Environments that 
exhibit all
>four of the ACID properties (Atomicity, Concurrency, 
Independence,
>Durability) [ACID].
>
>No multi-master replication system can avoid precluding 
support
>for ACID transactions. That is inherent in the definition. 
Nor
>does the C stand for Concurrency. Nor does the I usually 
stand
>for Independence.
>
>http://www.scism.sbu.ac.uk/georgeub/sharenotes/Wk10_Transacti
on_Concurrency/
>tsld003.htm
>
>ACID transactions are also impractical in a global 
distributed
>directory service even using single master - eg it is 
unscalable
>to maintain referential integrity between different servers.
>That is why they are excluded from the X.500 data model.
>
>The whole discussion of models is simply confusing and 
confused.
>It should be dropped as adding nothing to either requirements
>or understanding as demonstrated by the inaccurate 
definitions
>and references.

<rm>I thought you were originally arguing for this model
being supported.  Are you now changing your mind and saying
that this should be dropped?</rm>

>Instead definitions of concepts that are not already defined 
or well
>understood should be provided - eg granularity of 
operations, granularity of
>replication, concurrency, consistency, isolation, durability,
>serialization, convergence. In particular there is ongoing 
confusion about
>the
>whole concept of "loose consistency" and the fundamental 
distinction between
>the data model permitting different servers to have 
different versions
>of data with only "eventual" convergence, and changes 
inconsistent with
>the constraints of the data model.
>
>A meaningful replacement for G2 could then be something like 
this:
>
>"The protocols SHOULD NOT hinder standardization of 
extending the
>granularity
>of LDAP operations beyond a single entry."

<rm>That sentence is not acceptable to me because
(a) I don't see what it has to do with replication and
(b) I'm not sure what it really means.</rm>

>
>URP would not meet such a more reasonable requirement since 
it does
>not respect the granularity of operations being even at the
>attribute level, let alone the entry level.
>
>Other similar requirements would also be appropriate, eg:
>
>"The protocols SHOULD NOT hinder standardization of signed 
operations"

<rm>Did I miss signed operations becoming standard (I might
have)?  I believe its optional even if it is standardized
and so would be captured under the item of supporting
LDAP options that needs to be added.  If it isn't
standardized, then it falls into the same bucket as the
ACL discussions.</rm>

>14) Limited Effort Consistency
>
>G1.  LDAP Replication MUST support models 2 (Eventual 
Consistency) and
>3 (Limited Effort Eventual Consistency) above.
>
>URP makes no attempt to support model 3 (any failure in its 
attempt to
>support model 2 does not constitute supporting model 3, 
which is
>aimed at entirely different design goals completely 
irrelevant
>to LDAP directory replication).

<rm>If folks want to use LDAP in a Model 3 situation
then it should be replicatable by LDUP, right?

[snip]</rm>

---

From: "Ryan Moats" <[email-masked]>
Date: Mon, 26 Feb 2001 14:54:42 -0600

Ah... more snipping and <rm2>...</rm2> this time...

BTW, these are my views and my coauthors are free to
disagree with me :-)

-----Original Message-----
From: Albert Langer [mailto:[email-masked]]
Subject: RE: Proposed requirements not met by current architecture and
URP proposals - part 2 (end)

[snip]...

[Albert]
Looking forward to your views on this and other issues
(whether in this "last call" on requirements context or
in relation to URP etc). I'm assuming the editors will be going
through all issues raised by anyone whether you (or
John) have expressed an opinion about them now or not.

<rm2>I think that's a fair assumption :-)</rm2>

the in-order
convergence.
send all
master applied
not a
at the expense
justify SM2,
explanation does
requirements.
without

[Albert]
The way I see it any sensible design for Single Master
would maintain strict ordering - both to meet the necessity
for some sort of conformance level and migration and
coexistance strategy for existing single master implementations
implied by the WG charter (which I believe should also be
spelled out as a requirement), and also to simplify
implementation and meet requirements G2 and MM1 for
performance.

However the functional requirements are covered by G2, MM1
and the implicit or missing requirement to support
Single Master meaningfully.

The conclusion that maintaining strict ordering is
the best design for meeting such requirements can
be drawn in the architecture and does not need to be
spelled out as a requirement. I try to be very
consistent about this distinction between design
choices and requirements. Perhaps I am overly
sensitive to it in reaction to John's persistent
campaign to present my objections to URP not
meeting the requirements as just a cover for my
having been so presumptious as to show that an
alternative design was possible by submitting MDCR
when I initially drew attention to the basic flaws in
URP and insisted that the lack of requirements
that made it obvious such a design would be
unacceptable necessitated review of the
allegedly "ready" requirements draft.

My items 6 and 8 raise the relevant requirements
that URP does not comply with. The design choices
that resulted in that are a design problem, not
a requirements problem.

At any rate I certainly agree with the design
choice reflected in SM2, so the comment is
essentially editorial and I will not pursue it.

Concerning MM7, you seem to be suggesting that
it merely does not impose a requirement for strict
ordering on the design for multi-master. That could
be achieved by simply omitting it and perhaps
moving the explanation words elsewhere into a more
explicit statement within SM2 that it does not
apply to multi-master.

<rm2>Maybe it could be omitted, but SM2 is single master
and doesn't apply to multi master, and I (for one) would
rather say something then have a void on ordering.  The
dependence on ordering is very close to making a choice
between two general classes of implementation: state-based
versus change log and that's something that the WG will not
be successful mandating and has explicitly kept open in the
past.</rm2>

In fact MM7 explicitly requires that the design
for multi-master require consumers to accept out
of order changes during a replication session,
instead of requiring suppliers to keep them in
order. That is very clearly a design choice as
to *how* to satisfy requirements for (robust)
eventual convergence.

<rm2>I disagree.  If you go back through the archives, you
will see a lot of discussion of state-based versus change log
implementations.  Requirement MM7 says that either approach
may be used, because the state-based system may never be able
to send changes in the correct order.</rm2>

I believe it is a bad design choice which
unnecessarily complicates implementation and
directly frustrates achieving the performance
requirements in G2 and MM1, without adding
anything to the robustness of convergence.

<rm2>That may be true, but that should be determined by the
marketplace</rm2>

An immediate consequence is that replication
sessions have to be restarted from the
beginning when the link fails whereas they
can continue from where they left of when
strict ordering is used (as correctly noted
in the architecture). As multi-master is
especially useful in situations with weak
network connectivity that could be quite
a significant impact on performance.

It is also much easier for each supplier to
maintain a single ordered index of what it is
sending to various consumers than for each
consumer to have to deal with updates out
of order. Any simplification of the supplier
(which I doubt exists anyway) is completely
negated by the fact that every supplier is also a
consumer in multi-master and that such
indexing has to be done for slave read only
single master servers at the periphery anyway.

<rm2>Again, you are now into implementation and others
may not agree with you, so the requirements have to leave
other possibilities open</rm2>

I find the "negative consequences" listed in B.2
for out of order replication convincing and
the "beneficial" situations unconvincing. The
examples given in the "beneficial" section
concerning schema and other critical policy
information being replicated out of order are
the most dangerous things to replicate out of
order as immediately pointed out in the
"negative consequences". The only remaining
example is for pushing a priority update
ahead of a bulk reload which refers to Single
Master where you then prohibit it.

<rm2>I do not claim to be wiser than every directory administrator
in every scenario, so B.2 is worded with that (at least in my mind)

[snip...]</rm2>

and not
recovery from a
satisfied by
the URP approach
supplier

[Albert]
I think we might be at cross purposes. I was not supporting Steve's
comment on P2:

<rm2>[snip]...

I thought that there was mail already agreeing to change P2 to be
implementation neutral in response to other comments.

http://www.imc.org/ietf-ldup/mail-archive/msg00884.html
</rm2>

Transactional
exhibit all
Independence,
support
Nor
stand
on_Concurrency/
distributed
unscalable
confused.
definitions

I have *always* said that model 2 is the only model relevant
to LDUP work and that multi-master cannot possibly support ACID
transactions. To understand why one could reject the wording
of that requirement in view of the definition given for
Transactional Consistency, simply click on the link that
was provided.

Speaking frankly both the requirement above and the fact that
you thought I ever supported it, is entirely due to you,
like many other members of the WG, not actually understanding
the issue.

<rm2>[snip...]

Well, I think that you may have some difficulty in convincing
the WG to drop model 1, but now I understand.  Also I point out
that Model 1 is not required, but that there was sufficient
consensus behind not closing the door on it permanently</rm2>

Consistency) and
attempt to
which is
irrelevant

[snip]</rm>

Only if you believe a WG that has not been chartered to do that work,
has not done any such work, and knows nothing about the issues
involved, should nevertheless be told it MUST do it before having
completed work it is supposed to know about.

I can only repeat my remarks at the link provided in the message
you are replying to:

***
Model 3 ("limited effort consistency") is needed for multi-master
replication among such a large number of independent nodes that it becomes
impossible to maintain globally consistent state information as to which
nodes are actually participating at any given time. There may also be other
situations in which it is needed (eg nodes that are not "servers" attempting
to provide a reliable service to "clients" but just trying to get a better
picture of the world around them, without storage capacity for global
information). Examples that come to mind with "servers" include the
replication of newsgroups and (distributed) email lists. Despite the best
efforts of "backbones" it is impractical to ensure that all nodes do receive
all updates so the feeds diverge due to batches not being collected within
the "limited best effort" criteria for purges. Model 2 is simply not capable
of scaling to the size of systems such as Usenet with currently available
technology on a multi-master basis. It is inherently confined to situations
where a relatively small number of "servers" (hundreds, perhaps thousands,
but not millions) are maintaining a common "backbone" for a much larger
number of "slave" servers and end user clients - such as for example LDAP
directories ;-)

<rm2>Ah, this is your opinion.  However, again I believe that I have
seen enough WG consensus to keep Model 3 on the books.  I think we
may have to agree to disagree here.

[snip]</rm2>

---

From: "Albert Langer" <[email-masked]>
Date: Tue, 27 Feb 2001 07:07:34 +1100

[Ryan]

Section E is indeed saying "the requirements are wrong,
- so URP should not be required to meet them".

Sections A to D are saying "URP does not meet the
requirement" with B and D saying "but it could be fixed
to do so" whereas A and C are saying "and it's core
design prevents it being fixed to do so - a different
approach will be necessary if the requirement is
adopted".

I tried to explain the categories at the start of

>http://www.imc.org/ietf-ldup/mail-archive/msg00891.html

As you have noticed, there are some passing references
to requirements issues elsewhere, but in section E
that is the focus:

Requirements Problem: - The requirement could not or should not be
				implemented. This
				is not intended as a catalog of requirements issues, but only
				those of which it might be said that URP and related parts of
				the architecture has failed to conform, or that conformance
				to it provides an explanation for not conforming to another
				requirement. The requirement should	be changed or removed,
				not the specification.

I was working through the requirements and URP and architecture
systematically in
response to John's request, looking for requirements that are not met by URP
and
architecture so that it could not be said later that any of them were
adopted
without having had notice of their impact on URP. In doing so I came across
several
that would have to be included in that list, but which I do not believe URP
or
the architecture should be expected to satisfy because the problem lies in
the
requirement rather than URP or the architecture. Also at least 1 where they
do
conform but doing so is unnecessary or undesirable and provides a
justification
for not meeting other requirements. My focus is still URP and as a result of
that I have not had time to deal with non-URP related requirements issues,
except incidentally.

Basically it seems to me that the requirements have now reached a point
where
it is possible to take them seriously as potentially being used to evaluate
whether other drafts do or do not meet those requirements and there are
indications that they will actually be used that way, unlike the version
before you and Rick were added to the editorial team - for which you should
both be congratulated (and suitably rewarded with various requirements
issues to resolve ;-)

and redundancy
further comment

[snip]</rm>

[Albert]
Well the comment is essentially editorial and as you are the
editors I see no need to pursue it.

[Albert]

>7) Missing

[...]

[snip]...</rm>

[Albert]
Agreed - section E below is where I am suggesting the
requirements are problem. All the other items above are saying there
is a meaningful requirement which URP and/or the architecture
are being required to implement but do not implement - as
requested by John.

* E) Requirements Problem *

(Here beginneth the requirements items ;-)

[...]

the in-order
convergence.
send all
master applied
not a
at the expense
justify SM2,
explanation does
requirements.
without

[Albert]

Concerning MM2, you seem to be suggesting that

In fact MM2 explicitly requires that the design

In general I believe priority updates are
adequately catered for by simply being able
to trigger a replication session and no
significant advantage would be obtained
by additional priority that could justify
the additional complexity and performance
loss etc.

and not
recovery from a
satisfied by
the URP approach
supplier

[Albert]

http://www.imc.org/ietf-ldup/mail-archive/msg00846.html

The first part of his argument does indeed ignore G3 and MM1:

"Not resending acknowledged updates to a consumer requires something like
two-phase commit for the replication session between the supplier and
consumer, which isn't part of the current protocol proposal, and doesn't
need to be since URP absorbs any duplicates."

Absorbing duplicates instead of avoiding them obviously does not
minimize either system or network resources.

But I do agree with his second point:

"Also, with the Update Vector,
it is the consumer that keeps track of the updates that have been received
rather than the supplier keeping track of the updates that have been sent."

I have probably given the wrong impression by saying "there is nothing
unsound in the URP approach". What I really meant was just the rest
of the sentence "of having the consumer track updates received
rather than the supplier tracking updates sent". I had already
raised the matter of URP not meeting G3 and MM1 as item 6 in the
message you are responding to - categorized as "fixable".

The bigger problem of course is that replication sessions have
to be restarted from scratch. That is imposed by your design
constraint on ordering - item 11 above.

Your current wording mandates the supplier keeping track of
what consumers have received as the *means* to support G3 and MM1.

There is no reason why the goals of G3 and MM1 cannot be achieved
with the consumers keeping track. The "pull model" where consumers
specify an Update Vector and suppliers send only those items the
consumer has thereby said they haven't seen looks perfectly
sound to me.

The requirements G3 and MM1 simply need to be implemented, not
reinforced by attempting to specify the design to implement
them in a requirements draft.

Transactional
exhibit all
Independence,
support
Nor
stand
on_Concurrency/
distributed
unscalable
confused.
definitions

I stated that multi-master replication cannot
possibly support ACID transactions as simply as possible
in my first positive exchange with you on the list:

***
I think something like this should be in a new requirements draft:

"Changes to an LDAP directory can be:

a) Locally available
b) Atomic
c) Never revoked by the directory service

Pick any two.

This draft provides an explanation of why a choice must be made between b)
and c) for directory uses that require multi-master replication to achieve
a). It explains what sort of directory uses do require a) and explores the
implications of the possible choices between b) and c) and various
combinations and options for achieving them. Input is requested from other
areas as to what requirements they have for existing and future directory
applications and operational uses of the directory that should affect the
architectural choices now being made by the WG."
***
http://www.imc.org/ietf-ldup/mail-archive/msg00573.html

I believe there would be substantially less of the obvious confusion
on these issues now, if that work had been done then.

I have in the past used the word "transactions" loosely to refer
to operations affecting more than one entry but certainly never
suggested that multi-master could support ACID transactions.

When I first joined the WG I took it for granted
that it would be understood by anyone working on multi-master
replication issues that it is not possible to support
all four elements of ACID transactions, and the WG had simply
made a wrong choice as to which proper subset to support and
what granularity of replication to support.

Now that I know better I intend to avoid the use of the
word "transactions" myself and am proposing that the
the requirement (which is certainly *not* what I ever
proposed or supported) be altered to something actually
meaningful as below.

or well
confusion about
distinction between
different versions
inconsistent with
extending the

[Albert]
Fair enough. By itself (b) establishes my wording
is not clear enough, which I will leave to the editors,
to fix. The current wording and definition is not unclear
but plain wrong and that is not because I or anyone else has
changed their mind about something - it's just wrong.

The fundamental data model of the directory treats entries
as the granularity of update operations. Leaving aside
extensions to delete a subtree etc, there is simply no way
to express an LDAP operation to modify two different
entries at once. This is a deliberate choice for scalability
because the two entries may be distributed to different
servers. Likewise, contrary to URP, when you change anything
you are changing an entry as a whole, not just a single
attribute or attribute value. Applications rely on that
model of an entry as a grouping of *related* attributes
and values with a common name.

Some work is being done on operations that apply a
set of related changes, sometimes referred to as
"transactions" and sometimes more accurately as
"grouped operations". These have a granularity of
more than one entry.

Just as replication standards could break atomicity and
impede access control standards if not done right, replication
standards could impede such work if not done right.

For a definition of "SHOULD NOT IMPEDE", see my MDCR
draft.

For more on what replication has to do with it, see:

http://www.imc.org/ietf-ldup/mail-archive/msg00569.html

it does
operations"

[Albert]
I'm not sure what bucket ACL discussions will fall into,
though I am sure about what hole a directory replication
protocol that cannot support standardized ACLs will
fall into.

My comment is not limited to things that *have* been
standardized. It is based on a general attitude of
"first, do no harm".

As suggested, it may well belong with a wider item:

"In general a review of existing and proposed LDAP extensions should
be undertaken as part of requirements analysis to determine what
avoidable problems could result from multi-master deployment so
that the architecture and design will take those into account
and try to minimize their impact. This could best be done by
publishing an explanation of the sort of issues likely to arise
in an initial requirements document and actively soliciting feedback."

Consistency) and
attempt to
which is
irrelevant

[snip]</rm>

***

For example the current LDUP architecture requires each node to maintain and
exchange information about the state of all other nodes, which adds a very
significant overhead to each replication session and severely limits
scalability. Perhaps that could be improved on within the scope of model 2,
but it is a non-trivial problem for which nobody in LDUP has come up with a
suggestion.

I believe there will be a growing demand for improved protocols meeting
requirements for "model 3" as the return to understanding the internet as a
"peer to peer" network gathers pace. Examples include "Freenet" and
"Groove", which could easily point to future peer to peer replication
systems reaching millions of replicating nodes but could not hope to
maintain more than "best effort" guarantees. Some may well involve the use
of LDAP clients and even exchange information analagous to directory
entries, though I cannot think of any example of work in that area. But if
and when they do, that is *all* they will have in common with LDUP work. The
design issues for "model 3" replication protocols has virtually nothing in
common with those for "model 2" that actually relates to replication. LDUP
has not been chartered to do that work, has not done any such work, and
knows nothing about the issues involved.

We have to provide a viable solution for replication of LDAP directories.
They clearly require eventual convergence and would be useless with "limited
effort" convergence since the inevitable divergence would negate the
benefits desired from actual needs for LDAP directories.
***

Chris commented that "And you point out one of those good reasons in the
next
paragraph that you wrote...." [see above]

Perhaps I was not sufficiently clear in saying "LDUP has not been chartered
to
do that work, has not done any such work, and knows nothing about the issues
involved."

But any further clarification would require going beyond merely being blunt.

---

From: "Albert Langer" <[email-masked]>
Date: Wed, 28 Feb 2001 04:52:36 +1100

[Albert]
the in-order
convergence.
[...]

[...]

[Albert]
I noticed that when I went through the archives on first
joining the WG and have just taken another look. This
has heightened my concern that MM7 be removed or replaced with
something more neutral.

There are 3 plausible design choices that the *architecture*
can adopt for implementations conforming to multi-master standards.

a) Consumers MUST accept out of order changes from suppliers.

b) Suppliers MUST send changes in order to consumers.

c) Both of the above.

There are no other plausible options for interoperability such as
"leave it to the administrator" - that is merely a restatement
of c). Otherwise no interoperability is possible.

The current wording of MM7 clearly mandates option a) or c) and
prohibits adoption of the simpler option b) alone.

"MM7. Multi-master conflict resolution MUST NOT depend on the in-order
arrival of changes at a replica to assure eventual convergence."

That is consistent with the design choice made in the current
architecture draft and may well be the final outcome. But it is
clearly a design choice, not a requirement.

The reason for it you mention, "because the state-based system
may never be able to send changes in the correct order" could
well reflect a valid non-functional requirement such as:

"Design choices MUST reasonably balance the convenience of
implementation for state based and log based systems".

I'm not sure whether that should be described as a "resource
constraint", "architectural constraint", "policy constraint"
or what, but I'm not disputing it's a practical reality for the
WG.

Making that explicit would also cover your related remark:

"The dependence on ordering is very close to making a choice
between two general classes of implementation: state-based
versus change log and that's something that the WG will not
be successful mandating and has explicitly kept open in the
past."

But even without such an explicit additional requirement,
the arguments you mention could certainly be legitimately raised in
discussing the architecture and would undoubtedly prove
decisive if they stand up to scrutiny.

What I noticed when reading the discussions the first time,
and have confirmed again, is that these arguments *were* decisive,
but were *not* subjected to much scrutiny. Every other argument
pointed to a significantly more efficient and simpler design
with mandatory ordering. In particular MM7 essentially achieves
the exact opposite of G2 and MM1 by requiring a restart for any
interrupted replication session and also far more frequent
complete resynchronizations.

In my view it is quite clear that state based implementations
would not be inconvenienced *at all* by adopting option b).
All they need is a simple index in CSN order and they will
have to maintain that index anyway to support SM2 or an
equivalent design choice not expressed as a requirement.

The evidence for that is Active Directory, which is clearly
a state based implementation and does use option b) with
a simple index.

AD appears to be substantially more efficient than Novell's
implementation. Unfortunately Microsoft has not actively
participated in working out a design for interoperability.
So they have not challenged any of the bad design choices
in LDUP, although they certainly know better,
or at least hold an opposite opinion as to which design
choice is better than the choice which has prevailed in LDUP so
far on this issue, but have never argued for it.

[...]

[...]

LDUP has no option but to make a choice between a), b)
and c). Even listing c) is stretching a point
simply because it is a logically possible choice.
Ultimately the real choice comes down to a) or b) and
adding administrator options to a protocol like this
conflicts with well known internet architectural
principles. But that of course is a design issue.

[...]

Exactly my point. Anything I say about this is an
argument about design, not requirements. Please
try very hard to find a wording that enables
such discussion to be deferred until design
documents are being considered in the light
of requirements, rather than making it essential
to settle the issue during the next "final"
call on requirements by putting a design
choice in a requirements document.

[...]

and not
recovery from a
satisfied by
the URP approach
supplier

[Albert]
I think we might be at cross purposes.[...]

<rm2>[snip]...

[Albert]
We were indeed at cross purposes. You appeared to be explicitly
arguing for the current text above. I am quite happy with
Steve's proposal (although simple deletion would be better):

***
1) The replication architecture SHOULD avoid sending to a consumer an
update that the consumer has already seen.

2) An update received by a consumer more than once MUST NOT produce a
different outcome than if the update were received exactly once.
***

The weakening to SHOULD in 1) does not authorize just deciding to send
duplicates anyway if a better approach is available, but permits
balancing the undesirability of that against other valid design
considerations that can be justified, rather than imposing the method
for meeting performance goals in the requirements document. Essentially
it leaves it up to the design to meet the other requirements while
demanding an explanation if the choices made result in any duplicates
at all. That seems unecessary to me, but it doesn't really matter.

The MUST NOT in 2) adds nothing but also does no harm.

Note that these two are consistent with a design change to eliminate
duplicates completely (which almost necessarily requires sending
in strict order), although not requiring such a change. But any such
design change *would* be prohibited by MM7.

Transactional
exhibit all
Independence,
<rm2>[snip...]

[...]

Consistency) and

[...]

[snip]</rm2>

[Albert]
Agreeing to disagree is fine. Problem is that you are 
"agreeing to agree", which has no place in a standards
process. Unlike others that may be unnecessary, these
2 are simply meaningless.

I'm not going to be the one in difficulty
convincing anybody about anything. You're the bunny that
committed to:

<rm5>If URP goes to last call without (at least for me)
acceptible statements as to why it doesn't meet certain 
requirements, I'll try to be first in line saying "uh, I don't
see that you've met requirement <fill in blank> nor do I see
a statement as to why", that's all.</rm5>

http://www.imc.org/ietf-ldup/mail-archive/msg00838.html

So what answer is any author supposed to give as to *why* they
haven't made any attempt to not preclude support for model 1
and have provided no support for model 3?

There is nothing surprising about agreeable people agreeing
to agree and so reaching "consensus" not to drop something
without any intention of actually doing anything about it.
But that isn't part of producing a standards track protocol
for it.

If you are editor of a requirements document in a
standards process, it is your job to keep such "agreeableness"
out of it and make sure it consists only of operationally
meaningful requirements that can be applied to determine
whether or not those requirements have been satisfied.

I'm very impressed with the fact that you and Rick have
been able to turn a document that was largely meaningless
into one which *can* be used to ask such questions. I
wouldn't have thought it was possible, given the starting
point and environment. It's a shame to spoil that by undermining
the effect of those questions with 2 questions that are meaningless.

"Consensus" is no excuse. You are the guys that are supposed
to report back and point it out when a supposed consensus
does not actually mean anything. If you get instructed to
insert or remove something by a WG decision, after having
advised against it, that's a different matter. No such
instruction can be given to you as editors, without putting
it to the WG mailing list (not a meeting or a memo from John without
having actually *asked* the WG), and a public determination
reached by the co-chairs that a "rough consensus" does exist,
after giving you the opportunity to justify your recommendation
and others the opportunity to comment and record their views
on the specific issue that the WG is taking a decision about.

This mailing list is part of the public record of the
IETF standards making process. It is an engineering
process for producing actual internet protocols.
"Rough consensus" is the means by which
it takes decisions between mutually exclusive
alternatives, not a means for avoiding them, or for
recording Johns views on various subjects from day
to day, or rubber stamping the "room consensus" at
a meeting.

No such WG instruction has been given to you on this or
any other matter. The point is you haven't advised against
it so you are clearly identified as the authors of 2
"requirements" that you know, or ought to know,
are meaningless.

Having said that, I agree to disagree ;-)

---

From: "Ryan Moats" <[email-masked]>
Date: Tue, 27 Feb 2001 12:39:08 -0600

<rm3>This time</rm>

-----Original Message-----
From: Albert Langer [mailto:[email-masked]]
Subject: RE: Proposed requirements not met by current architecture and

[Albert]
the in-order
convergence.
[...]

[...]

[Albert]

c) Both of the above.

<rm3>Actually, I don't see (b) being precluded by MM7.  In fact
(b) has always been assumed at least by me and so I was aiming
for (c) in the  sense that while servers MUST send changes in
order, it is quite possible due to topology that a client will
receive changes in a different order ("in order arrival").

If you are assuming that (b) guarantees that (a) will never
occur, you'll have a task convincing me.</rm3>

<rm3>I think we had done that.  Obviously, you disagree</rm3>

{...}

<rm3>Well, that's not my reading of MM7, but I'll think about it
some more</rm3>

[Albert]

<rm3>Sorry, you aren't going to tie me in knots. That's for
the to answer, when they go to last call. I'm not going to
burn brain cells worrying about it :-)</rm3>

<rm3>It is necessary to "agree to agree".  This goes on at
all levels (i.e. between the editors, inside the WG, inside
the IETF). My point is that, just because I can't see how
something can be done, doesn't mean that there somebody out
there isn't smarter than I am and can solve the problem and
so MHO is that the door should remain open. If the WG 
consensus changes (and I don't think it has), then you are
right that it should change.

Further, if the collective folks doing the architecture or
protocol or some other *standards* track document decide they
can't do it, they have to say why they can't do it.  I may
not like the reason for why they can't do it, but if the WG
consensus is that the reason is acceptable, I have to accept
it.</rm3>

---

From: "Albert Langer" <[email-masked]>
Date: Wed, 28 Feb 2001 22:11:09 +1100

[Albert]
(continuing discussion from:
 "RE: Proposed requirements not met by current architecture and URP
proposals - part 2 (end)"
http://www.imc.org/ietf-ldup/mail-archive/msg00916.html)

the in-order
convergence.
[...]

c) Both of the above.

[Albert]
1. Your assumption that (b) could be taken for granted is natural.

We are of course talking about a single replication session in a context
where it is well understood that globally throughout the replication
network, changes from different originator DSAs will arrive at particular
replicas interleaved arbitrarily due to transitive replication with
bilateral rather than globally imposed schedules and arbitrary dynamic
changes to the actual links connecting DSAs as a replication network.

It is necessary to design for that even for an enterprise level, let
alone global internet level replication standard. The architecture is
designed for that, and the fundamental approach used with Update Vectors
and Purge Vectors looks sound. Perhaps that should be noted as a real
achievement of the LDUP WG, that is likely to be "won and recorded"
when we do get to architecture and verify that this fundamental
architecture is indeed capable of meeting the relevant requirements.

Within that fundamental framework 3 major design issues for propagation
of changes arise:

1.1 Should the layer responsible for propagating reports about changes
from originator DSAs for eventual convergence of the entire replication
set via the replication network deliver every report to every DSA, so
that each applies the same deterministic procedures to the same reports
(necessarily interleaved) and can easily arrive at eventual convergence
so long as those procedures are able to cope with the interleaving, or
should it be mixed together with the layer responsible for conflict
resolution within each DSA in order to achieve some performance
benefit with deterministic procedures that do not need to see all
the reports and may generate new reports summarizing reports
discarded during transitive propagation.

A natural assumption is that one would choose a layered approach
in line with commonly accepted architectural principles.

Nevertheless, the current draft does not, and I will be raising
that as a design issue. It is not a requirements issue because
it is sometimes appropriate to mix layers in order to better
meet specific performance requirements. (A requirement not
to create transient visible directory states that are not the result
of any DUA operation could be appropriate but seems to be
covered by other requirements and any sound approach to
meeting them).

1.2 Should the report propagation mechanism ensure that the
reports from each originator DSA are delivered to every other
DSA in the same sequence that they were generated by each
originator (interleaved of course with the sequences from
other originators).

A natural assumption is that it should, since that fits
naturally with the purge vector mechanism for preventing
the unbounded growth of meta-data and generally simplifies
things.

Nevertheless, the current architecture does not. Again it is a
design issue, not requirements.

1.3 Should the protocol between each pair of
DSAs that have a replication agreement in either
direction transmit reports from the supplier to
the consumer in the same sequence that they
were received or originated by the supplier.

A natural assumption is that it should. I have
never heard before of an application protocol that
takes a strictly or partially ordered batch of
messages that it has to deliver all of, and allows
them to be delivered in arbitrary order on a
simple link between two nodes.

Nevertheless, what you took for granted is not
in fact the case and will be raised as a design
issue, because it is not a requirements issue.

Incidentally, considerable progress was made towards
resolving these issues in discussions between
Steve, Zachary and me in the thread on "10 Report
Propagation Sequence", as can be seen from this
comment by Steve:

***
There would seem to be four choices:
1) unordered within session,
2) ordered on CSN per replica ID,
3) ordered on CSN,
4) ordered on supplier receipt order.

Those vendors wanting unordered transmission should speak up now.
I'm happy with 1, 2 or 3.
***
http://www.imc.org/ietf-ldup/mail-archive/msg00693.html

Nobody has spoken up wanting unordered transmission but
neither have either the architecture or URP drafts changed
from currently permitting unordered transmission. Nor have
the WG chairs done anything to put that issue to the
WG for a decision.

So contrary to your assumption, MM7 was not viewed by
authors of architecture and URP drafts, (who have raised
no objection to it), as implying c) rather than a).

2. Your concern that b) alone does not guarantee a) will
never occur is adequately covered by the requirement for
eventual convergence repeated 3 times and further
reinforced by the editors consensus to replace P2 with
two requirements that include:

***

What more can you possibly need to ensure that your concern
MUST be met by any proposal in design discussions for b) alone?

What can you achieve by forcing such discussion to take place
in the next final call on requirements?

{...}

[Albert]
Fine. When thinking about it, please think in terms of other
peoples assumptions and the way it might be read by others
rather than the way you read it.

BTW that {...} above was a suberb piece of editing to avoid
my coat-trailing about review of WG progress and IETF procedures
from intruding into the discussion you want to have now on
requirements ;-)

Similar editing of your own text such as MM7 can ensure design
issues do not delay finalizing requirements.

[...]

* Model 1 and Model 3 *
[...]

<rm3>Sorry, you aren't going to tie me in knots. That's for
the[m] to answer, when they go to last call. I'm not going to
burn brain cells worrying about it :-)</rm3>

Fair enough. But you already mentioned you thought I supported
the requirement not to impede model 1. By stating a requirement
to not impede ACID transactions that is logically impossible for
multi-master you are obscuring real concerns that can be
expressed as real requirements that people don't have to burn
up brain cells about.

Simply add, even if you won't replace:

a) For single master environments, replication standards
SHOULD NOT impede future work on ACID transactions.

(That seems to be Kurt's concern, though I'm still not
sure)

b) For both single master and multi-master environments,
replication standards SHOULD NOT impede future work on
grouped operations affecting more than 1 entry.

* Agreeing to Agree *

I agree with all of that. The normal process involves a
continuous give and take with the smaller teams responsible
for editing particular documents as an integral whole,
submitting their drafts for regular review in which
outstanding issues can either get resolved by WG decisions
or left for resolution later.

By actually reviewing comments and responding to them
seriously, either in the list or in new drafts, you
and Rick are now moving the WG towards resolution on
requirements. When others follow that example, or the
chairs start putting unresolved issues that have been
debated enough up for WG decision in the list, similar
progress will be made on other drafts.

BTW please consider your point about "just because I can't
see how something can be done", in relation to MM7 ;-)

But on the issue of whether we are working on the
X.500/LDAP model ("model 2"), or something else, that
ought to have been resolved by the WG charter in the
first place.

Sure the editors working on architecture and other
drafts ought to have responded that they can't do it
re model 1 and ought to have asked what you are
asking them to do re model 3.

But you are the authors of a document requiring that they
MUST support model 2, with a reference to
Xerox Clearinghouse that isn't about it at all,
and *you* need to be able to explain why anyone
MUST do that.

Chris wrote:

***
It would not be listed as a MUST at
this stage of revising the requirements document if the authors (some of the
top LDAP experts in the world) and many other members of the working group
didn't know of a good reason for it being stated as that strong of a
requirement.
***

http://www.imc.org/ietf-ldup/mail-archive/msg00738.html

As he is a WG co-chair (or was - I haven't seen any sign of him for quite
a while) I took the trouble of spending some time looking for information
about Xerox Clearinghouse before deciding that it would not be useful
to respond to such messages from Chris.

I was already aware that the paper actually referenced by the authors,
about Bayou, was relevant to model 2, not model 3, since Bayou supports
"eventual consistency" (though with all the extra complications for
application semantics that are avoided by the much "lighter" atomicity
requirement in LDAP).

The "many other members of the WG" who are supposed to know why we
MUST work on model 3 are under no obligation to respond.

But the authors of the requirements draft have been cited by a WG
co-chair as people who "know of a good reason for it".

You are one of those authors, and this is a WG last call on
a requirement *you* have proposed. Before expecting
anyone *else* to "burn brain cells" explaining why they haven't done
it, *you* (ie the editors) have some obligation to explain why *you*
say they MUST do it.

I've said all I want to say about these issues for now. It's up to the
editors whether further discussion is needed in the next "final"
call.

I suggest putting any further explanations in the accompanying
material for the next draft, rather than continuing this now.