Subject: [apps-discuss] tail -f over HTTP -- what am I missing?

From: Nico Williams <[email-masked]>
Date: Thu, 24 Apr 2014 12:59:06 -0500

Suppose you have an append-only, variable-size entry log file, but a new
one could be renamed into place.  Clearly the logfile's strong ETag would
have to change with each entry appended to it, but then how's the client to
know if a GET of a range of it will return new entries from the same log
file as before or possibly garbage from a new one -- garbage because the
requested range is meaningless given the change in the log file's identity.

Ideally there would be something like ETag that corresponds to resource
identity (st_dev+st_ino in POSIX terms) but doesn't change when the
resource's size changes.  If multiple ETags can sensibly be sent for one
resource then sending a weak etag and a strong etag in should suffice. Is
that permitted?  If so a tail -f client could periodically attempt to GET
additions using ranges and If-Match the weak etag.

Now, reading RFC2616 sections 14.24 and 13.3.3 makes me sad: 13.3.3 says
that clients can't use weak validators with range requests, and 14.24 says
that If-Match must use strong validators.  This seems to preclude sensible
tail -f behavior over HTTP!

What am I missing?  I can't be the first to want this, but I'm not finding
categorical answers to how to tail -f over HTTP...

Of course, getting servers to sensibly generate appropriate etags is not
trivial, and less so when using clustered services or third party server
farms.  And a tail -f client would have to be pretty smart, not just a dumb
GETter.  It might be easier to synthesize per-update URIs and a "latest"
resource, at the price of doing two requests to fetch updates.  Indeed,
that's that I'll be doing for the application I need this for since,
thankfully, it is easy in this one case to identify complete
"transactions".  But this approach doesn't generalize.

IF there was a standard way to tail -f over HTTP then servers might well
support sensible (clusterable) weak/strong etag computations for resources
known to be log-like.

Nico
--

---

From: Martin Thomson <[email-masked]>
Date: Thu, 24 Apr 2014 11:09:52 -0700

> categorical answers to how to tail -f over HTTP...

Have you considered the possibility that different log entries could
be their own resources?  Or is that too hard?  It means application
logic rather than range requests to pull and compose entries, but
that's not unusual.  You could use Link: rel=prev or rel=next with
either ranges or entries to help with crawling the log.

The case of copying a new file over seems like it wants a layer of
indirection.  If you don't like 3xx, Content-Location might work.

Just spitballin':

GET /full-log

200 OK
Link: </2014-05-01/13..>;rel=next
Content-Location: /2014-05-01/1..12

{12 records}

GET /2014-05-01/13..

200 OK
Link: </2014-05-01/1..12>;rel=prev
Link: <2014-05-01/16..>;rel=next
Content-Location: /2014-05-01/13..15

{3 records}

---

From: Julian Reschke <[email-masked]>
Date: Thu, 24 Apr 2014 20:17:46 +0200

On 24.04.2014 19:59, Nico Williams wrote:
 > ...
> that clients can't use weak validators with range requests, and 14.24
> says that If-Match must use strong validators.  This seems to preclude
> sensible tail -f behavior over HTTP!
> ...

Why are you looking at RFC2616 anyway? Did you miss that the new HTTP 
specs have been approved in February?

Best regards, Julian

---

From: Nico Williams <[email-masked]>
Date: Thu, 24 Apr 2014 14:01:31 -0500

> specs have been approved in February?

OK, I have those in front of me.  It's still not obvious.  Can you tell me
what the answer is?

Nico
--

---

From: Nico Williams <[email-masked]>
Date: Thu, 24 Apr 2014 14:42:44 -0500

On Thursday, April 24, 2014, Martin Thomson <[email-masked]>
wrote:

> On 24 April 2014 10:59, Nico Williams <[email-masked]<javascript:;>>
> wrote:

Fortunately in my very specific case I can change the app to generate log
file increments, each as its own file/resource, chaining back with links
embedded in the resource.  So yes, that's my solution for now.

But that's a lot of work for a generic server to with plain old log files
in a generic way!  In fact, it can't be done in a generic way.  Yet in a
sense a log file is a static resource -- it should be trivial and fast to
serve it and incremental additions to it.  Is that possible?  How??  I
don't see it.  I can't be the first person to need such a thing, but I
can't find how to do it.

The Unix tail(1) -f is trivial, it reads to EOF, then polls for size
changes and reads new data when it shows up.  An HTTP equivalent ought to
work, but since this is akin to building a filesystem we need something
akin to st_dev/st_ino to detect whether the likely-to-be-modified resource
is still the same resource in the sense of being the same file rather than
having the same contents (which is what strong ETAgs provide).  The obvious
thing to use is weak ETags, but as I noted, that's forbidden (I haven't yet
parsed enough of the new HTTP/1.1 specs to know if that's fixed, but
since server support is what matters...).

> indirection.  If you don't like 3xx, Content-Location might work

Yes, that's what. i"m doing -- but only because it happens to be feasible
in this case.

But I want to know how to do this in a general and -naturally-
high-performance manner.  I'm asking a question about the standards, not my
particular situation.

Nico
--

---

From: Martin Thomson <[email-masked]>
Date: Thu, 24 Apr 2014 13:05:45 -0700

On 24 April 2014 12:42, Nico Williams <[email-masked]> wrote:
> The Unix tail(1) -f is trivial, it reads to EOF, then polls for size changes
> and reads new data when it shows up.  An HTTP equivalent ought to work

tail(1) fails to notice changes to the file behind its read horizon,
right?  An HTTP version should work, if you only care for the same
sort of loose guarantee.  Just don't use If-Range and make a blind
range request.

I think that Content-Location (or redirects) is the best way to
address the issue of swap-out.

---

From: Nico Williams <[email-masked]>
Date: Thu, 24 Apr 2014 15:18:21 -0500

Hmm,  i think I found a way: use GET requests with If-Modified-Since and
Range (starting at the last content-length, the old EOF) but NOT
If-None-Match, then check the ETag in the response if successful, if the
ETag does not match the one the client knew from earlier, then the range
was invalid and the client must abort or fetch the new resource whole.

This works with weak etags only, of course.  If a resource had two etags
(is that possible?? that's not clear) use whichever one was weak.   If the
resource has no weak etags then it is not meant to be used in a tail -f
way.  For POSIX systems the weak etag should be constructed from the log
file's st_dev and st_ino.

The only problem with this is that the client might sometimes
wastefully read a range of a new file, but that's exceedingly unlikely.

Not every server I've checked can build such weak etags, but that's alright.

Note: I still don't know if a resource may have more than one etag.

Nico
--

---

From: Julian Reschke <[email-masked]>
Date: Thu, 24 Apr 2014 23:36:04 +0200

On 24.04.2014 21:01, Nico Williams wrote:
> On Thursday, April 24, 2014, Julian Reschke <[email-masked]
> <mailto:[email-masked]>> wrote:
>
>     Why are you looking at RFC2616 anyway? Did you miss that the new
>     HTTP specs have been approved in February?
>
>
> OK, I have those in front of me.  It's still not obvious.  Can you tell
> me what the answer is?

No, I just wanted to make sure you are looking at the latest and greatest...

---

From: Mark Nottingham <[email-masked]>
Date: Mon, 28 Apr 2014 14:08:03 +1000

Range requests really weren’t built for tailing a stream of data. Do something in the application, e.g., using URIs, prev/next links, etc.

Cheers,

On 25 Apr 2014, at 3:59 am, Nico Williams <[email-masked]> wrote:

> Suppose you have an append-only, variable-size entry log file, but a new one could be renamed into place.  Clearly the logfile's strong ETag would have to change with each entry appended to it, but then how's the client to know if a GET of a range of it will return new entries from the same log file as before or possibly garbage from a new one -- garbage because the requested range is meaningless given the change in the log file's identity.
> 
> Ideally there would be something like ETag that corresponds to resource identity (st_dev+st_ino in POSIX terms) but doesn't change when the resource's size changes.  If multiple ETags can sensibly be sent for one resource then sending a weak etag and a strong etag in should suffice. Is that permitted?  If so a tail -f client could periodically attempt to GET additions using ranges and If-Match the weak etag. 
> 
> Now, reading RFC2616 sections 14.24 and 13.3.3 makes me sad: 13.3.3 says that clients can't use weak validators with range requests, and 14.24 says that If-Match must use strong validators.  This seems to preclude sensible tail -f behavior over HTTP!
> 
> What am I missing?  I can't be the first to want this, but I'm not finding categorical answers to how to tail -f over HTTP...
> 
> Of course, getting servers to sensibly generate appropriate etags is not trivial, and less so when using clustered services or third party server farms.  And a tail -f client would have to be pretty smart, not just a dumb GETter.  It might be easier to synthesize per-update URIs and a "latest" resource, at the price of doing two requests to fetch updates.  Indeed, that's that I'll be doing for the application I need this for since, thankfully, it is easy in this one case to identify complete "transactions".  But this approach doesn't generalize.
> 
> IF there was a standard way to tail -f over HTTP then servers might well support sensible (clusterable) weak/strong etag computations for resources known to be log-like.
> 
> Nico
> -- 
> 
> 
> _______________________________________________
> apps-discuss mailing list
> [email-masked]
> https://www.ietf.org/mailman/listinfo/apps-discuss

--
Mark Nottingham   http://www.mnot.net/

---

From: Nico Williams <[email-masked]>
Date: Tue, 29 Apr 2014 10:30:07 -0500

> Range requests really weren’t built for tailing a stream of data. Do something in the application, e.g., using URIs, prev/next links, etc.

What were Range requests built for?

In my application I have something of a logical transaction that I can
use to group log entries into individual never-appended-to files.  So
that's what I'm doing now and we can stop considering my immediate
needs.

However, my current approach and all the suggestions to treat log
entries as resources don't generalize.  There really are append-only
log files out there which one might want to tail -f.  Just as tail -f
makes sense as a Unix application, I think it makes sense as an HTTP
application.

If my proposed construction of tail -f is invalid/incorrect, I'd like
a correction.

If a tail -f cannot be built with HTTP/1.1 or 2.0 as they are now, I'd
like confirmation.

If a tail -f should not be built at all with any version of HTTP, I'd
like to hear arguments as to why.

Maybe I should move this to the HTTPbis list?

Nico
--

---

From: Julian Reschke <[email-masked]>
Date: Tue, 29 Apr 2014 18:17:06 +0200

On 2014-04-24 19:59, Nico Williams wrote:
> one could be renamed into place.  Clearly the logfile's strong ETag
> would have to change with each entry appended to it, but then how's the
> client to know if a GET of a range of it will return new entries from
> the same log file as before or possibly garbage from a new one
> -- garbage because the requested range is meaningless given the change
> in the log file's identity.
>

In WebDAV there's a resource-id property: 
<http://greenbytes.de/tech/webdav/rfc5842.html#rfc.section.3.1>. What's 
missing is the ability to use it in a conditional request header field, 
although it might be possible to extends WebDAV's "If" header field.

Best regards, Julian

---

From: Mark Nottingham <[email-masked]>
Date: Wed, 30 Apr 2014 15:16:24 +1000

> What were Range requests built for?

Mostly, interacting with PDFs, AIUI.

> makes sense as a Unix application, I think it makes sense as an HTTP

Sure. It’s just that you’re trying to layer your application in using a *generic* mechanism (ranges), when something more application-specific (using URIs, media types, links, etc.) is more appropriate.

> Maybe I should move this to the HTTPbis list?

I think the appropriate folks are probably in both places.

Cheers,

---

From: Larry Masinter <[email-masked]>
Date: Wed, 30 Apr 2014 14:55:36 +0000

> Mostly, interacting with PDFs, AIUI.

http://tools.ietf.org/html/draft-ietf-http-range-retrieval-00
notes that restarting partial downloads was also an original motivation. 

For some reason,  I had expected the web to migrate to using
compound documents
http://en.wikipedia.org/wiki/Compound_document

but to avoid having to download the entire package before you 
could start  rendering.  

I think MPEG-Dash designers considered using range retrieval
but decided it was too unreliably available (many proxies
not doing range retrieval right).

Larry

---

From: Nico Williams <[email-masked]>
Date: Wed, 30 Apr 2014 15:43:51 -0500

> notes that restarting partial downloads was also an original motivation.

Append-only logs seem like a good fit for this: downloads are always
partial!  :) (until the log is closed anyways)

> but to avoid having to download the entire package before you

Log files aren't compound documents.  I keep hearing that I should
treat log files as collections of resources, as if having millions of
100-200 byte resources scales, as if having a file per log entry would
be a good idea.  There's a reason we have log files and not log
directories.  Nor is it feasible to assign URIs to individual log
entries in a log file -- or if it is we'd have to put offsets into
those URIs and end up doing range requests in a different way.

> not doing range retrieval right).

Thanks, that's useful information.

Nico
--

---

From: Nico Williams <[email-masked]>
Date: Wed, 30 Apr 2014 15:46:55 -0500

> Sure. It’s just that you’re trying to layer your application in using a *generic* mechanism (ranges), when something more application-specific (using URIs, media types, links, etc.) is more appropr…

My application is taken care of.  But surely there's a general need
for this sort of thing.

Who here hasn't run "tail -f some-log-file"?  Of those who have, who
has had to ssh to some server to get to that log file?

> I think the appropriate folks are probably in both places.

I thought so.  Good to know.

Nico
--

---

From: Martin Thomson <[email-masked]>
Date: Wed, 30 Apr 2014 15:27:42 -0700

On 30 April 2014 13:43, Nico Williams <[email-masked]> wrote:
> as if having millions of
> 100-200 byte resources scales

Works for me...

---

From: Mark Nottingham <[email-masked]>
Date: Thu, 1 May 2014 09:29:47 +1000

… and they don’t have to be “millions of 100-200 byte resources”; the granularity of your pages is up to you.

Cheers,