Subject: RE: Rethinking scope on publish, was Re: [Simple] PUBLISH: atomicity problem

From: [email-masked]
Date: Wed, 16 Jul 2003 22:07:27 +0300

> -----Original Message-----
> From: ext Paul Kyzivat [mailto:[email-masked]]
> Sent: Wednesday, July 16, 2003 1:19 AM
> To: Jonathan Rosenberg
> Cc: Robert Sparks; Niemi Aki (NMP/Helsinki); [email-masked]
> Subject: Re: Rethinking scope on publish, was Re: [Simple] PUBLISH:
> atomicity problem
> 
> 
> I generally like this idea, but maybe it is still too complex.
> 
> Why do we need the Pstream-ID? Simply say each publisher must choose 
> unique tuple ids and leave it at that.
> 
> Regarding how to set up blocking of the ones you don't want, 
> instead of 
> pstream, how about simply having each publisher include its 
> identity in 
> each tuple, as an attribute? This could be filtered out for 
> most users.

tuple is presence specific. I was thinking of a time-stamp type header (for presence, it probably carries the same info as in the time-stamp element). But then Jonathan mentioned deleting a tuple.

Hisham

> 
> 	Paul
> 
> Jonathan Rosenberg wrote:
> > I was about to post a note on agreeing with Aki.  I was 
> also going to 
> > say that we might want the body to also be the atomic unit, 
> so that when 
> > you publish a tuple, the body is an XML fragment for the 
> tuple. Then, it 
> > ocurred to me that we were really far down the complexity 
> ramp here. So, 
> > I thought about what we could do to really back out of this mess 
> > entirely. Here is my idea.
> > 
> > In the beginning, PUBLISH was easy. Each PUA published its 
> own view of 
> > the presence of the presentity. It had no knowledge of 
> other PUA. It 
> > published a complete presence document which represented 
> what it thought 
> > was the full presence of the presentity. Indeed, if there 
> was only one 
> > PUA, it would be the full presence of the presentity. There 
> was no way 
> > for one PUA to overwrite the published data from another. 
> The tuple IDs 
> > didn't need to have cross-PUA scope. No naming conventions 
> or anything. 
> > The compositor could combine these and modify them as it 
> chose. Each 
> > represented a unique and different piece of info about the 
> presentity. A 
> > useful bit of information was this "Pstream-ID", which uniquely 
> > identified the publisher. You can think of the Pstream-ID 
> as a way of 
> > scoping the tuple IDs. Each tuple ID is defined only within 
> the scope of 
> > a Pstream-ID, and each publisher has to have a unique PStream-ID.
> > 
> > It all got complicated when we decided to try and solve a bigger 
> > problem, where PUA could override information about other PUA. That 
> > introduced all kinds of problems. We needed to define the atomic 
> > information of published presence. We need to resolve tuple 
> naming. We 
> > need to worry about this issue about whether a publish has 
> one or more 
> > tuples. We needed to add Etags and If-Modified-Since. We 
> need to worry 
> > about how to figure out what it means for the system to 
> "wait for human 
> > input" in cases where thats not possible. We don't know how 
> to do that yet.
> > 
> > Perhaps we should reconsider whether we want to solve this 
> problem this 
> > way. We have only one use case - when I leave my PC at the 
> office with a 
> > presence state that is wrong, and I want to fix it at home. 
> Well, this 
> > is a nice feature, but is it worth all of the complexity at 
> this point? 
> > There are other ways to deal with it. In fact, I can think of two 
> > alternatives:
> > 
> > 1. You could use the auth policy to simply disable the PC's 
> data from 
> > being placed in notifications, and then publish a separate 
> tuple (i.e., 
> > new piece of independent information) from home. Thats not 
> as good (PC 
> > still refreshes useless state), but its lots simpler than 
> the route we 
> > are on.
> > 
> > 2. You could design a system which only allowed one PUA at 
> a time (akin 
> > to the current model in AIM and YIM). This would manifest itself as 
> > composition policy. The compositor would only use the 
> published document 
> > from the PUA which started publishing most recently - that 
> is, when its 
> > Pstream-ID first appeared. This would also achieve the effect of 
> > disabling the work PC's state when you arrive at home.
> > 
> > 
> > So, to be concrete, I would propose the following:
> > 
> > 
> > 1. Publications carry complete presence documents, representing the 
> > complete presence state of the PUA as far as it knows.
> > 
> > 2. Publishers are identified by a unique PStream-ID, chosen by the 
> > publisher. Copying someone elses Pstream-ID is not allowed, 
> and behavior 
> > in this case is not defined.
> > 
> > 3. There is no ability to overwrite someone elses presence 
> document or 
> > their tuples. Each publisher publishes independent information.
> > 
> > 4. We eliminate the Etag, If-Modified-Since, and atomicity/segment 
> > discussions from Publish.
> > 
> > 5. Compositors can combine the publsihed information 
> however they like. 
> > There is no need to carry through tuple IDs from publication to 
> > notifications.
> > 
> > 6. If you want to block a tuple from getting into a notify, 
> you identify 
> > it by a distinguishing property rather than by tuple ID.
> > 
> > 
> > I believe this approach works nicely for other things too. 
> In the case 
> > of winfo, where you have lots of presence servers, each 
> publishing winfo 
> > for the subscriptions they have, its the same. Each publisher (a 
> > presence server), publishes the complete view of the state 
> - its own 
> > subscriptions. There is no overlap between servers. The compositor 
> > combines these together using a basic policy (combine subscriptions 
> > together that are for the same presentity), and thats it.
> > 
> > I believe it works for mwi too.
> > 
> > I really feel that it is important that we make some 
> simplifications so 
> > that we can make rapid progress here. I don't think we lose 
> much since 
> > we can still get the desired feature; we just get it in a 
> different way.
> > 
> > Thoughts?
> > 
> > -Jonathan R.
> > 
> > Robert Sparks wrote:
> > 
> >> Another way to state this conclusion is that we need to report
> >> success or failure of publishing each atomic element in an 
> independent
> >> fashion. One quick and easy way to achieve that property 
> is to allow
> >> only one atomic element per PUBLISH request.
> >>
> >> I think I heard another way to do this mentioned during 
> the meeting,
> >> where we move reporting the success/failure and any feedback from
> >> the attempt to publish each tuple into the message instead 
> of trying
> >> to capture it all on the status line. For example (I think this is 
> >> what I heard suggested during the meeting), for presence, we could
> >> publish an entire pidf document with several tuples and in 
> the presence
> >> response return an xml document with one element per tuple 
> discussing
> >> the disposition of attempting to publish that tuple.
> >>
> >> Something along the lines of the following (an example to 
> convey the
> >> concept, not a proposal of a syntax)
> >> <publish-results>
> >>   <tuple id="023cfdsf3">
> >>    <success etag="39009sdf" expires="1800"/>
> >>   </tuple>
> >>   <tuple id="990429834">
> >>    <failure/>
> >>   </tuple>
> >> </publish>
> >>
> >> If we were to pursue such an optimization, it might make sense to
> >> look at how to specify a separate etag value in the 
> publish (currently
> >> what we put in an If-Match header) for each atom in the published
> >> document.
> >>
> >> RjS
> >>
> >> On Mon, 2003-07-14 at 19:58, [email-masked] wrote:
> >>
> >>> All,
> >>>
> >>> I'll try to summarize the issues w.r.t. the atomicity of 
> publication 
> >>> here in an attempt to converge the discussion towards a 
> solution to 
> >>> this problem. BTW, I realized that my slides in the 
> SIMPLE session 
> >>> were perhaps not crystal clear on what the problem at hand really 
> >>> was, so my apologies for that.
> >>>
> >>> The core problem is this:
> >>> EPA X sends a PUBLISH with {a, b, c}, after which EPA Y 
> publishes {a, 
> >>> b}. When EPA X tries to refresh its publication of {a, b, c}, the 
> >>> request fails because of a collision, resulting in the EPA X 
> >>> quitting. EPA Y still refreshes its publication of {a, b}, which 
> >>> means {c} expires and goes away. As a result, the 
> publication process 
> >>> has lost information of {c}. This is unacceptable, and with the 
> >>> current versioning mechanism, this can't be remedied by composer 
> >>> policy, for example.
> >>>
> >>> Instead, to get around the above problem, what you send 
> in a PUBLISH 
> >>> needs to match the atomic element of the published 
> information. So 
> >>> each PUBLISH needs to contain a single tuple, because the 
> versioning 
> >>> and expiration all work on the single atomic element level anyway.
> >>>
> >>> This has the apparent drawback in that the PUA has to 
> wait for a 200 
> >>> OK after each PUBLISHed tuple before it can PUBLISH the next one. 
> >>> This occurs upon every refresh cycle, and in systems with high 
> >>> latency, may become a major issue in terms of performance.
> >>>
> >>> Ways to get around this inefficiency:
> >>>     - Remove the restriction on overlapping requests, i.e., allow 
> >>> "pipelining"
> >>>       of PUBLISH requests
> >>>     - Treat the publisher of each tuple as a separate EPA, and 
> >>> therefore not
> >>>       subject to this restriction in the first place (seems like 
> >>> cheating...)
> >>> In conclusion, my proposal is to go with having always a 
> single tuple 
> >>> in a PUBLISH, and allow overlapping PUBLISH requests. To my 
> >>> knowledge, this would fulfill all the related 
> requirements and solve 
> >>> the issue.
> >>>
> >>> Thoughts?
> >>>
> >>> Cheers,
> >>> Aki
> >>>
> >>> _______________________________________________
> >>> Simple mailing list
> >>> [email-masked]
> >>> https://www1.ietf.org/mailman/listinfo/simple
> >>
> >>
> >>
> >>
> >> _______________________________________________
> >> Simple mailing list
> >> [email-masked]
> >> https://www1.ietf.org/mailman/listinfo/simple
> >>
> > 
> 
> 
> 
> _______________________________________________
> Simple mailing list
> [email-masked]
> https://www1.ietf.org/mailman/listinfo/simple
> 

_______________________________________________
Simple mailing list
[email-masked]
https://www1.ietf.org/mailman/listinfo/simple

---

From: [email-masked]
Date: Thu, 17 Jul 2003 12:04:07 +0300

> From: ext Jonathan Rosenberg [mailto:[email-masked]]
 > Sent: 15 July, 2003 23:38
 > To: Henning Schulzrinne
 > Cc: [email-masked]
 > Henning Schulzrinne wrote:
 > 
 > > This does seem much simpler. I wonder if combining this 
 > with an explicit 
 > > remove operation - "drop this Pstream/tuple-id tuple" - 
 > would avoid the 
 > > rather round-about way of cleaning up dead state noted 
 > below, without 
 > > having to worry about atomicity, last-modified-issues and 
 > unique naming.
 > 
 > I don't follow. Thats exactly what we were doing before. The home PC 
 > will send your "drop this pstream/tuple-id" in a publish (I assume 
 > that is what you mean), and then the work PC refreshes, and now we 
 > have the conflict all over again.
 > 
 > One other point on my proposal. One unresolved issue which 
 > we had was, 
 > if you want to delete a tuple using Expires:0, what 
 > information in the 
 > publish identified the tuple? In the proposal I am making, its easy. 
 > The Publish has an Expires:0 and a Pstream-ID header. The published 
 > document from that pstream is deleted.

Well, in the current draft, this would be done using the server provided etag, and Expires:0. The pstream-ID is really just another way of doing etags.

Cheers,
Aki

 > 
 > -Jonathan R.
 > 
 > -- 
 > Jonathan D. Rosenberg, Ph.D.                600 Lanidex Plaza
 > Chief Technology Officer                    Parsippany, NJ 07054-2711
 > dynamicsoft
 > [email-masked]                     FAX:   (973) 952-5050
 > http://www.jdrosen.net                      PHONE: (973) 952-5000
 > http://www.dynamicsoft.com
 > 
 >

---

From: [email-masked]
Date: Thu, 17 Jul 2003 12:04:07 +0300

Cheers,
Aki

---

From: [email-masked]
Date: Thu, 17 Jul 2003 12:04:02 +0300

Hi Jonathan,

In general, I like the proposal of simplifying things. 

If I understand the proposal you're making correctly, you want to remove all of the current discussion on segmenting presence state, and have each PUA publish their full view. Also, instead of using server-provided etags, you want a publication to be identified by a client-generated pstream-ID.

Firstly, I agree on removing segmentation, and the whole package that comes with it (collision detection, overriding, tuple-ID naming conventions etc.) As you say, this just complicates things unnecessaríly, and leaving these details to composer and publication authorization policy makes a lot of sense.

But about using PStream-ID instead of etags, I don't think it saves us a lot compared to the current usage of Etags. By having the server generate these identifiers (etags), their uniqueness within a presentity is guaranteed. With PStream-IDs, the EPA has to do some work in order to generate an ID that is guaranteed to be unique. Admittedly, this isn't a huge effort, but compared to having the server assign identifiers simply by incrementing a counter, may be slightly more complex. It anyway introduces the chance of colliding PStream-IDs, which we would have to deal with one way or another.

Also, with PStream-ID, you'd still need an error response in case no such PStrean-ID exists in the ESC. So all in all, we save the task of defining/implementing the If-Match header field, but need more to handle the PStream-ID generation/collision issues.

So my counter-proposal would be to simplify along the lines you described, removing all of the segmentation things,  but still keep the Etags, If-Match headers. The only change in their usage would be to remove the collision detection altogether, so that the only case when a refresh/deletion would fail with a 412 would be if the publication had already expired (i.e., the etag no longer existed).

Cheers,
Aki

 > Sent: 15 July, 2003 22:42
 > To: Robert Sparks
 > Cc: Niemi Aki (NMP/Helsinki); [email-masked]
 > Subject: Rethinking scope on publish, was Re: [Simple] PUBLISH:
 > atomicity problem
 > 
 > 
 > I was about to post a note on agreeing with Aki.  I was also 
 > going to 
 > say that we might want the body to also be the atomic unit, so that 
 > when you publish a tuple, the body is an XML fragment for the tuple. 
 > Then, it ocurred to me that we were really far down the complexity 
 > ramp here. So, I thought about what we could do to really 
 > back out of 
 > this mess entirely. Here is my idea.
 > 
 > the presence of the presentity. It had no knowledge of other PUA. It 
 > published a complete presence document which represented what it 
 > thought was the full presence of the presentity. Indeed, if 
 > there was 
 > only one PUA, it would be the full presence of the presentity. There 
 > was no way for one PUA to overwrite the published data from another. 
 > The tuple IDs didn't need to have cross-PUA scope. No naming 
 > conventions or anything. The compositor could combine these 
 > and modify 
 > them as it chose. Each represented a unique and different piece of 
 > info about the presentity. A useful bit of information was this 
 > "Pstream-ID", which uniquely identified the publisher. You can think 
 > of the Pstream-ID as a way of scoping the tuple IDs. Each 
 > tuple ID is 
 > defined only within the scope of a Pstream-ID, and each 
 > publisher has 
 > to have a unique PStream-ID.

I'm not sure what type of temporal scope you are suggesting for the stream-ID. Is it very long term, like a device-ID? Or is it only persistent across a specific publication, over its refreshes, deletion, etc.?

If the only reason for stream-ID was to scope the tuple-IDs, then whatever uniqueness property we want for the stream-ID, can equally well be applied to the tuple-IDs, making the actual stream-ID not needed at all.

 > about how to figure out what it means for the system to "wait for 
 > human input" in cases where thats not possible. We don't know how to 
 > do that yet.

I agree, the overriding requirement lead us to the problems in atomicity, and tuple-ID naming.
 
 > Perhaps we should reconsider whether we want to solve this problem 
 > this way. We have only one use case - when I leave my PC at 
 > the office 
 > with a presence state that is wrong, and I want to fix it at home. 
 > Well, this is a nice feature, but is it worth all of the 
 > complexity at 
 > this point? There are other ways to deal with it. In fact, I 
 > can think 
 > of two alternatives:
 > 
 > being placed in notifications, and then publish a separate tuple 
 > (i.e., new piece of independent information) from home. Thats not as 
 > good (PC still refreshes useless state), but its lots 
 > simpler than the 
 > route we are on.

But how would you actually do that? If the only way a publisher can be identified (PStream-ID) is generated by the publisher itself, how can one set the appropriate authorization policy for disabling those publications?
 
 > 2. You could design a system which only allowed one PUA at a time 
 > (akin to the current model in AIM and YIM). This would 
 > manifest itself 
 > as composition policy. The compositor would only use the published 
 > document from the PUA which started publishing most recently - that 
 > is, when its Pstream-ID first appeared. This would also achieve the 
 > effect of disabling the work PC's state when you arrive at home.

 > publisher. Copying someone elses Pstream-ID is not allowed, and 
 > behavior in this case is not defined.
 > 
 > 5. Compositors can combine the publsihed information however they 
 > like. There is no need to carry through tuple IDs from 
 > publication to 
 > notifications.
 > 
 > 6. If you want to block a tuple from getting into a notify, you 
 > identify it by a distinguishing property rather than by tuple ID.
 > 
 > 
 > of winfo, where you have lots of presence servers, each publishing 
 > winfo for the subscriptions they have, its the same. Each 
 > publisher (a 
 > presence server), publishes the complete view of the state - its own 
 > I really feel that it is important that we make some simplifications 
 > so that we can make rapid progress here. I don't think we lose much 
 > since we can still get the desired feature; we just get it in a 
 > different way.
 > 
 > Thoughts?
 > 
 > -Jonathan R.
 > 
 > Robert Sparks wrote:
 > 
 > publication here in an attempt to converge the discussion 
 > towards a solution to this problem. BTW, I realized that my 
 > slides in the SIMPLE session were perhaps not crystal clear 
 > on what the problem at hand really was, so my apologies for that.
 > >>
 > publishes {a, b}. When EPA X tries to refresh its 
 > publication of {a, b, c}, the request fails because of a 
 > collision, resulting in the EPA X quitting. EPA Y still 
 > refreshes its publication of {a, b}, which means {c} expires 
 > and goes away. As a result, the publication process has lost 
 > information of {c}. This is unacceptable, and with the 
 > current versioning mechanism, this can't be remedied by 
 > composer policy, for example.
 > >>
 > >>Instead, to get around the above problem, what you send in 
 > a PUBLISH needs to match the atomic element of the published 
 > information. So each PUBLISH needs to contain a single 
 > tuple, because the versioning and expiration all work on the 
 > single atomic element level anyway.
 > >>
 > >>This has the apparent drawback in that the PUA has to wait 
 > for a 200 OK after each PUBLISHed tuple before it can 
 > PUBLISH the next one. This occurs upon every refresh cycle, 
 > and in systems with high latency, may become a major issue 
 > in terms of performance.
 > >>
 > >>	- Remove the restriction on overlapping requests, i.e., 
 > allow "pipelining"
 > >>	  of PUBLISH requests
 > >>	- Treat the publisher of each tuple as a separate EPA, 
 > and therefore not
 > >>	  subject to this restriction in the first place (seems 
 > like cheating...) 
 > >>
 > single tuple in a PUBLISH, and allow overlapping PUBLISH 
 > requests. To my knowledge, this would fulfill all the 
 > related requirements and solve the issue.
 > >>
 > >>Thoughts?
 > >>
 > >>Cheers,
 > >>Aki
 > >>

---

From: [email-masked]
Date: Thu, 17 Jul 2003 12:10:53 +0300

> Sent: Thursday, July 17, 2003 11:18 AM
> To: Khartabil Hisham (NMP/Helsinki)
> Cc: [email-masked]; [email-masked]; Niemi Aki
> (NMP/Helsinki); [email-masked]
> inline.
> [email-masked] wrote:
> 
> > 
> >> -----Original Message----- From: ext Paul Kyzivat
> >> [mailto:[email-masked]] Sent: Wednesday, July 16, 2003 1:19
> >> AM To: Jonathan Rosenberg Cc: Robert Sparks; Niemi Aki
> >> (NMP/Helsinki); [email-masked] Subject: Re: Rethinking scope on
> >> publish, was Re: [Simple] PUBLISH: atomicity problem
> >> 
> >> 
> >> Why do we need the Pstream-ID? Simply say each publisher must
> >> choose unique tuple ids and leave it at that.
> >> 
> >> instead of pstream, how about simply having each publisher
> >> include its identity in each tuple, as an attribute? This could
> >> be filtered out for most users.
> > 
> > 
> > tuple is presence specific. I was thinking of a time-stamp type
> > header (for presence, it probably carries the same info as in the
> > time-stamp element). But then Jonathan mentioned deleting a tuple.
> > 
> 
> My proposal is that deletion of a tuple is done using the 
> authorization stuff. Its not a publishing issue.

In an earlier post, you proposed using Expires: 0 and pstream header to delete a tuple. I was pointing out that having a time-stamp type header was sufficient enough to identify who published the most fresh information. Then I realised it was actually not sufficient enough due to your proposal to use the pstream header for deleting.

I was also objecting to having something in the tuple to identify the most recent publish since a tuple is presence specific.

> 
> The ID header is needed for soft-state. If the information being 
> published is represented by the content of the PUBLISH request, you 
> need an identifier in there to know when a PUBLISH updates/refreshes 
> something published previously, or its a new PUA. You cannot rely on 
> the From field or credentials for this disambiguation, since I may 
> have multiple devices acting on my behalf, all publishing for me, all 
> using my credentials and identity information.

Silly suggestion, but can we use contact-header for that?

> 
> I think that pstream-ID was rejected previously because we were ALSO 
> using tuple as semantically meaningful identifiers in publication 
> processing. I am proposing that this not be the case. The tuple IDs 
> only have meaning within the context of the published stream 
> (which is 
> pretty much the same semantic they have in a notification stream).
> 
> This also allows us to draw a clean line between the process of 
> composition, which is data-type specific, and this PUBLISH mechanism, 
> which is meant to not be specific to the data type. The "output" of 
> the publish mechanism, towards the compositor, is a series of N 
> published documents for a resource. The publish processor doesnt know 
> what these documents are, or what they represent. It only knows about 
> this pstream ID as a way of tagging them, so it can do the soft-state 
> management necessary.

That's exactly what I said on the mike in SIMPLE meeting, therefore I agree with you. The reply from one of the chairs was that we have to define the behaviour of the compositor anyway.

Regards,
Hisham

> 
> Also, consider further that deletion of published documents cannot 
> rely on some kind of data-type specific identifier, like a tuple. 
> Thats because the PUBLISH message with Expires:0 won't have any 
> content. So, you NEED to have an identifier in the header for this to 
> work.
> 
> -Jonathan R.
> 
> 
> --

---

From: [email-masked]
Date: Thu, 17 Jul 2003 12:38:37 +0300

> From: ext [email-masked] [mailto:[email-masked]]
> Sent: Thursday, July 17, 2003 12:04 PM
> To: [email-masked]; [email-masked]
> Cc: [email-masked]
> Subject: RE: Rethinking scope on publish, was Re: [Simple] PUBLISH:
> atomicity problem
> 
> 
> 
> 
> Well, in the current draft, this would be done using the 
> server provided etag, and Expires:0. The pstream-ID is really 
> just another way of doing etags.

So lets keep etags.

Hisham

---

From: [email-masked]
Date: Thu, 17 Jul 2003 12:38:37 +0300

So lets keep etags.

Hisham