Subject: [apps-discuss] Media type for lists of changed URIs?

From: Jan Algermissen <[email-masked]>
Date: Wed, 8 Jan 2014 15:46:08 +0100

Hi all,

I am about to work on a media type for transferring lists of URIs of resource that have changed to an 'observer' of a collection of resources.

It just occurred to me that something like that might already exist in the context of cache invalidation protocols.?

Jan

---

From: Bjoern Hoehrmann <[email-masked]>
Date: Wed, 08 Jan 2014 16:06:13 +0100

* Jan Algermissen wrote:
>I am about to work on a media type for transferring lists of URIs of 
>resource that have changed to an 'observer' of a collection of 
>resources.
>
>It just occurred to me that something like that might already exist in 
>the context of cache invalidation protocols.?

http://tools.ietf.org/html/rfc2483 defines `text/uri-list`.
-- 
Björn Höhrmann · mailto:[email-masked] · http://bjoern.hoehrmann.de
Am Badedeich 7 · Telefon: +49(0)160/4415681 · http://www.bjoernsworld.de
25899 Dagebüll · PGP Pub. KeyID: 0xA4357E78 · http://www.websitedev.de/

---

From: Darrel Miller <[email-masked]>
Date: Wed, 8 Jan 2014 10:08:28 -0500

Argggh gmail.  Forgot to cc the list.

---------- Forwarded message ----------
From: Darrel Miller <[email-masked]>

Could you instead define a link relation like rel="changed" and then
return text/uri-list ?

Darrel

On Wed, Jan 8, 2014 at 9:46 AM, Jan Algermissen
<[email-masked]> wrote:
> _______________________________________________
> apps-discuss mailing list
> [email-masked]
> https://www.ietf.org/mailman/listinfo/apps-discuss

---

From: Jan Algermissen <[email-masked]>
Date: Wed, 8 Jan 2014 16:10:26 +0100

> Argggh gmail.  Forgot to cc the list.

Well then, here's my reply again :-)

> Could you instead define a link relation like rel="changed" and then

Yes, exactly where I am at.

However, I'd like to remove the burdon of timing issues from the "protocol" and instead use linked paged (rel=prev-archive) for clients to process backwards until their last remembered timestamp.

Hence, I'd need something like

timestamp uri
timestamp uri

and this is where I wrote the last email :-)

Jan

---

From: mike amundsen <[email-masked]>
Date: Wed, 8 Jan 2014 10:19:45 -0500

timestamp thing is a wrinkle.

maybe Atom w/ tombstones?
http://tools.ietf.org/html/rfc6721

mamund
+1.859.757.1449
skype: mca.amundsen
http://amundsen.com/blog/
http://twitter.com/mamund
https://github.com/mamund
http://linkedin.com/in/mamund

---------- Forwarded message ----------
From: Jan Algermissen <[email-masked]>
Date: Wed, Jan 8, 2014 at 9:57 AM
Subject: Re: [apps-discuss] Media type for lists of changed URIs?
To: mike amundsen <[email-masked]>

On 08.01.2014, at 15:55, mike amundsen <[email-masked]> wrote:

> text/uri-list ?

[Same reply as to Darrel :-)] for you:

However, I'd like to remove the burdon of timing issues from the "protocol"
and instead use linked paged (rel=prev-archive) for clients to process
backwards until their last remembered timestamp.

timestamp uri
timestamp uri

Jan

> On Jan 8, 2014 9:46 AM, "Jan Algermissen" <[email-masked]>
> I am about to work on a media type for transferring lists of URIs of
resource that have changed to an 'observer' of a collection of resources.

---

From: Darrel Miller <[email-masked]>
Date: Wed, 8 Jan 2014 10:29:42 -0500

Jan,

I'm sure you could shoehorn it into a bigger media type like atom or
activity stream, but I haven't seen anything that fits that specific
need.  I'm not sure I follow your prev-archive scenario.  Do you want
a user to be able to ask, "what were the last-updated-dates on these
resources, yesterday, and the day before, etc"?

Darrel

>> It just occurred to me that something like that might already exist in the
>> context of cache invalidation protocols.?
>>
>> Jan
>>
>>

---

From: Jan Algermissen <[email-masked]>
Date: Wed, 8 Jan 2014 16:37:38 +0100

> I'm sure you could shoehorn it into a bigger media type like atom or

Yeah ... that's the big solution[1] (and [2] for the cooler kids), including maybe delta-updates. Wanted the less-overhead solution first.

> resources, yesterday, and the day before, etc"?

I want to avoid the situation where the client needs to understand the URI structure of the 'update-event' pages, e.g.

/items/changes/2013/01/01/01
/items/changes/2013/01/01/...
/items/changes/2013/01/01/24
/items/changes/2013/02/01/01
/items/changes/2013/02/01/...
/items/changes/2013/02/01/24

Because then I need more spec-parts (URI template vars, for example).

Instead, the client could start with a /changes/current (uncacheable) page and then proceed back in time through archived (cacheable) pages.

Jan

Credit where credit belongs:
[1] http://www.infoq.com/presentations/robinson-restful-enterprise
[2] http://roy.gbiv.com/untangled/2008/paper-tigers-and-hidden-dragons

---

From: "Markus Lanthaler" <[email-masked]>
Date: Wed, 8 Jan 2014 16:55:45 +0100

On Wednesday, January 08, 2014 4:38 PM, Jan Algermissen wrote:
> I want to avoid the situation where the client needs to understand the
> URI structure of the 'update-event' pages, e.g.
> 

Why do you need the timestamp in the URL *list*? Can't you just iterate
until you find a URL the client already knows? You could also use HTTP's
Last-Modified header to get the timestamp (but that would probably be very
inefficient).

--
Markus Lanthaler
@markuslanthaler

---

From: Jan Algermissen <[email-masked]>
Date: Wed, 8 Jan 2014 17:13:12 +0100

On 08.01.2014, at 16:55, "Markus Lanthaler" <[email-masked]> wrote:

> until you find a URL the client already knows?

Resources can be updated anytime, so I might miss something if I just look for URIs.
Jan

> You could also use HTTP's

---

From: Mark Baker <[email-masked]>
Date: Wed, 8 Jan 2014 11:58:06 -0500

> Hence, I'd need something like

I hear text/csv is making a comeback!

http://www.w3.org/2013/csvw/wiki/Main_Page

One could also argue that for a given set of headers, a +csv suffix
might also be appropriate :P

---

From: "Markus Lanthaler" <[email-masked]>
Date: Wed, 8 Jan 2014 18:00:27 +0100

On Wednesday, January 08, 2014 5:13 PM, Jan Algermissen wrote:
> On 08.01.2014, at 16:55, "Markus Lanthaler" wrote:
> 
> >> I want to avoid the situation where the client needs to understand
> >> the URI structure of the 'update-event' pages, e.g.
> >>
> > Why do you need the timestamp in the URL *list*? Can't you just
> > iterate until you find a URL the client already knows?
> 

Oh I see, I think I misunderstood what you are trying to build... not sure I
got it now but wouldn't sitemaps [1] offer exactly what you need then (apart
from not being a proper media type).

[1] http://www.sitemaps.org/protocol.html

--
Markus Lanthaler
@markuslanthaler

---

From: Jan Algermissen <[email-masked]>
Date: Wed, 8 Jan 2014 18:11:30 +0100

> I hear text/csv is making a comeback!

Great! Love CSV

> might also be appropriate :P

Definitely ... makes for better extensibility rules.

Thanks.

Jan

---

From: Jan Algermissen <[email-masked]>
Date: Wed, 8 Jan 2014 18:12:56 +0100

> from not being a proper media type).

Good point, thanks. Bit chatty, though.

Jan

---

From: Larry Masinter <[email-masked]>
Date: Thu, 9 Jan 2014 01:15:25 +0000

It sounds like this is an application that needs some extensibility, and you would be better off with

application/json

since the content doesn't really appear in a context where having a more specific media type is useful.

Larry
--
http://larry.masinter.net

From: apps-discuss [mailto:[email-masked]] On Behalf Of mike amundsen
Sent: Wednesday, January 08, 2014 7:20 AM
To: IETF Apps Discuss
Subject: [apps-discuss] Fwd: Media type for lists of changed URIs?

mamund
+1.859.757.1449
skype: mca.amundsen

---------- Forwarded message ----------
From: Jan Algermissen <[email-masked]<mailto:[email-masked]>>
Date: Wed, Jan 8, 2014 at 9:57 AM
Subject: Re: [apps-discuss] Media type for lists of changed URIs?
To: mike amundsen <[email-masked]<mailto:[email-masked]>>

timestamp uri
timestamp uri

Jan

> On Jan 8, 2014 9:46 AM, "Jan Algermissen" <[email-masked]<mailto:[email-masked]>> wrote:
> Hi all,
>
> [email-masked]<mailto:[email-masked]>

---

From: Tim Bray <[email-masked]>
Date: Wed, 8 Jan 2014 17:21:06 -0800

What Larry said, I’d do this in JSON and invest my efforts in the
application semantics.

On Wed, Jan 8, 2014 at 5:15 PM, Larry Masinter <[email-masked]> wrote:

> since the content doesn’t really appear in a context where having a more
> specific media type is useful.
>
>
>
> Larry
>
> --
>
> *From:* apps-discuss [mailto:[email-masked]] *On Behalf Of
> *mike amundsen
> *Sent:* Wednesday, January 08, 2014 7:20 AM
> *To:* IETF Apps Discuss
> *Subject:* [apps-discuss] Fwd: Media type for lists of changed URIs?
>
>
>
> From: *Jan Algermissen* <[email-masked]>
> However, I'd like to remove the burdon of timing issues from the
> "protocol" and instead use linked paged (rel=prev-archive) for clients to
> process backwards until their last remembered timestamp.
>

---

From: Jan Algermissen <[email-masked]>
Date: Thu, 9 Jan 2014 02:24:08 +0100

On 09.01.2014, at 02:21, Tim Bray <[email-masked]> wrote:

> What Larry said, I’d do this in JSON and invest my efforts in the application semantics.

Interesting - where in REST do you put application semantics if not in the media type?

Jan

> since the content doesn’t really appear in a context where having a more specific media type is useful.
> 
>  
> 
> Larry
> 
> --
> 
> > On Jan 8, 2014 9:46 AM, "Jan Algermissen" <[email-masked]> wrote:
> > Hi all,
> >

---

From: Darrel Miller <[email-masked]>
Date: Wed, 8 Jan 2014 20:26:41 -0500

> Interesting - where in REST do you put application semantics if not in the media type?

I was seconds away from pressing send with the same question.

Darrel

---

From: Tim Bray <[email-masked]>
Date: Wed, 8 Jan 2014 17:38:36 -0800

application/url-delta+json

---

From: Jan Algermissen <[email-masked]>
Date: Thu, 9 Jan 2014 08:00:48 +0100

> application/url-delta+json

Ah, I mistook you guys to mean application/json.

And +1 to the extensibility.

Jan

> 
> 
> On Wed, Jan 8, 2014 at 5:26 PM, Darrel Miller <[email-masked]> wrote:

---

From: Jan Algermissen <[email-masked]>
Date: Thu, 9 Jan 2014 15:11:41 +0100

On 09.01.2014, at 14:37, Graham Klyne <[email-masked]> wrote:

> Hi,
> 
> Maybe ResourceSync (http://www.openarchives.org/rs/0.9.1/resourcesync) has something for you?

Wow - basically what I am up to. How did I manage to miss it. 

Only thing that's missing, AFAICS, and which  I'll eventually need is the put/patch payload in the change list because in my scenario the number of updates and frequency is too high to justify individual GETs to sync. 

Will take a deeper look anyhow.

> 
> (The data formats used are based on SiteMap/XML rather than JSON as recommended by others in this thread.  

I actuall like XML quite a bit, still.

> Maybe unhelpful is the current lack of a separate MIME type?)
> 

Well, .... yup :-)

Jan

> #g
> --
> 
> On 08/01/2014 14:46, Jan Algermissen wrote:
>> Hi all,
>>

---

From: Jan Algermissen <[email-masked]>
Date: Thu, 9 Jan 2014 15:23:49 +0100

> Maybe ResourceSync (http://www.openarchives.org/rs/0.9.1/resourcesync) has something for you?

Any plans to register the link relation type resourceSync[1] with IANA?

Jan

[1] http://www.openarchives.org/rs/0.9.1/resourcesync#disco_links

> 
> (The data formats used are based on SiteMap/XML rather than JSON as recommended by others in this thread.  Maybe unhelpful is the current lack of a separate MIME type?)
> 
> #g
> --
>

---

From: Jan Algermissen <[email-masked]>
Date: Thu, 9 Jan 2014 15:39:59 +0100

On 09.01.2014, at 15:29, Graham Klyne <[email-masked]> wrote:

> On 09/01/2014 14:11, Jan Algermissen wrote:
>> 
> There are provisions for payload and patch-style operations through resource/change dumps and related-resource links respectively.

Ok, must have missed that at first glance. Either way, pretty exciting you ended up alost exactly where I did.

I have a question regarding a real practical problem from the eCommerce domain:

When eShops send product information to price comparison search engines there is the inherent problem of latency between price updates in the eShop and the re-publishing+ingestion to the shopping engine. If customers find higher prices in the shop, than on the site of the shopping engine, legal fees apply in some countries[1].

Have you ever thought of (or know about plans to) having the major shopping engines adopt resource sync to mitigate that problem? Would lead to a relief of quite some headaches of quite some people I must say.

Given there is sitemap in the mix, I thought Google might have a hand in it somehow.

Jan

[1] German federal court for example ruled the prices on the site may not be higher because "the customer can expect the systems to be in sync". Which shows an astonishing lack of understanding of the technical domain being ruled upon, IMHO :-)

> 
> Also, the group is currently discussing PubSubHubbub-based notifications to reduce polling load for learning about updates.  The initial draft for that proposal should be public very soon (hopefully days).
> 
>> 
>>> 
> I just posed that as a question to the ResourceSync working group.  There's a public discussion list at https://groups.google.com/forum/?fromgroups#!forum/resourcesync
> 
> This would probably be a good time to get involved with the discussions :)
> 
> #g
> --
> 
>>>

---

From: Graham Klyne <[email-masked]>
Date: Thu, 09 Jan 2014 14:37:12 +0000

> Any plans to register the link relation type resourceSync[1] with IANA?

Yes, I believe so (though I can't find any registration drafts right now).

"Relation types other than the ones listed above can be used in the Resourcesync 
framework. Valid relation types must be registered in the IANA Link Relation 
Type Registry or expressed as URIs as specified in RFC 5988, Sec. 4.2. The 
document [Relation Types Used in the ResourceSync Framework] attempts to provide 
an up-to-date overview." -- 
http://www.openarchives.org/rs/0.9.1/resourcesync#DocumentFormats

See also: http://www.openarchives.org/rs/0.9.1/resourcesync#LinkRelRes

#g
--

---

From: Graham Klyne <[email-masked]>
Date: Thu, 09 Jan 2014 14:29:07 +0000

> Will take a deeper look anyhow.

There are provisions for payload and patch-style operations through 
resource/change dumps and related-resource links respectively.

Also, the group is currently discussing PubSubHubbub-based notifications to 
reduce polling load for learning about updates.  The initial draft for that 
proposal should be public very soon (hopefully days).

>> Maybe unhelpful is the current lack of a separate MIME type?)

I just posed that as a question to the ResourceSync working group.  There's a 
public discussion list at 
https://groups.google.com/forum/?fromgroups#!forum/resourcesync

#g
--

---

From: Graham Klyne <[email-masked]>
Date: Thu, 09 Jan 2014 15:12:05 +0000

> Given there is sitemap in the mix, I thought Google might have a hand in it somehow.

There are not, to my knowledge, any explicit e-commerce involvements, or plans 
for such, but that's not a deliberate exclusion.  Also, no direct involvement 
with Google AFAIK, but the ResourceSync group did have some discussions with 
some Sitemap people about ensuring consistency of the extensions introduced. 
(Sitemap isn't *only* a Google thing:  cf. http://www.sitemaps.org.)

But the design has attempted to be applicable to web resources in the broadest 
sense.

Specifically, on the issue of synchronization, the design is intended to allow 
the Destination of a ResourceSync change list to be able to determine if it has 
the latest published update.  Some considerable discussion was given to timing 
issues and having confidence that updates could not go silently missing.  But, 
of course, the implementations would be ultimately responsible for ensuring 
changes are pushed out and used consistently.  (E.g. of the Source chooses to 
delay publication of an update, then there's no way for the Destination to know 
about it.)

#g
--

---

From: Graham Klyne <[email-masked]>
Date: Thu, 09 Jan 2014 13:37:21 +0000

Hi,

Maybe ResourceSync (http://www.openarchives.org/rs/0.9.1/resourcesync) has 
something for you?

(The data formats used are based on SiteMap/XML rather than JSON as recommended 
by others in this thread.  Maybe unhelpful is the current lack of a separate 
MIME type?)

#g
--