Subject: Paging, Feed History, etc.

From: Mark Nottingham <[email-masked]>
Date: Tue, 6 Jun 2006 17:54:14 -0700

I've been talking to a number of people face-to-face about Feed  
History and the use cases for it, and I've come to a position where I  
believe there are two major use cases out there for putting together  
multiple feeds to form one big, virtual feed.

1) So-called "incremental" feeds, where changes are happening at the  
"front end" of the feed, while less recent entries are pretty much  
static. If a change is required in an old entry, either a new revised  
entry with the same ID, or something like a tombstone is required.  
Publishers can easily make stable archives of old entries available,  
and clients wish to take advantage of caching, etc. to avoid re- 
transferring old entries again and again. A high degree of fidelity  
may be required; e.g., it should be possible to accurately  
reconstruct the entire state of the feed with no missed entries.  
E.g., a blog feed, but this could also be seen as an "event feed",  
where the entries are changes to the state of the underlying resources.

2) So-called "paging" feeds, where the entries are often the results  
to a query, being paged through in groups so as to not overwhelm the  
server and/or communications link. Entries may be arbitrarily added,  
deleted and reordered. Clients expect to access what the portions  
they need in relatively quick succession, and do not require absolute  
fidelity. E.g., OpenSearch query results. The entries in the feed  
directly correspond to the underlying resources, 1-to-1.

These are very different things. Incremental feeds require that the  
server keep state around per-feed, which isn't viable for something  
like query results, but fine for a blog. Paging feeds can lose  
entries (e.g., if http://example.org/index.page?page=1 refers to  
http://example.org/index.atom?page=2, page 2's contents can change  
between the two fetches), which is OK for some applications, and not  
for others.

As such, I'm pretty much convinced that they need to be dealt with  
separately.

Originally, I proposed that Feed History use "prev-archive" link  
relations, but many people pushed back on that in favour of the more  
generic "previous" and "next".

Given the above, I'd like to see if anyone would still object to  
having separate relation sets for incremental feeds ("prev-archive"  
and friends) and paging feeds ("previous", "next" and friends).

If people think that's a good idea, I can prepare a new draft that  
attempts to address both. The intent would be to be compatible with  
current usage by OpenSearch, GData, etc., while giving people the  
option to use something more reliable when necessary.

Thoughts?

--
Mark Nottingham     http://www.mnot.net/

---

From: John Panzer <[email-masked]>
Date: Tue, 06 Jun 2006 18:55:56 -0700

> option to use something more reliable when necessary.

+1. I think this is worthwhile[1] and I'd have no objections, even if 
this requires changing our implementation of previous/next.  Having 
attempted an implementation, I believe there's a substantial risk of 
confusion between incremental and paging relations and different names 
would help a lot.  Also, I can easily see us wanting to implement both 
incremental feeds and paging feeds in the same application at least.

-John

[1] Full disclosure: I'm one of the people who talked to Mark face to 
face about this.

---

From: "M. David Peterson" <[email-masked]>
Date: Tue, 6 Jun 2006 20:24:41 -0600

Definite +1

-- 
<M:D/>

M. David Peterson
http://www.xsltblog.com/

---

From: "Sylvain Hellegouarch" <[email-masked]>
Date: Wed, 7 Jun 2006 08:01:02 +0100 (BST)

> Definite +1

+1 as well.

---

From: "James Holderness" <[email-masked]>
Date: Wed, 7 Jun 2006 12:26:34 +0100

I'm going to reserve judgement until I see what exactly it is you're 
proposing, but I'm not particularly keen to change our existing 
implementation.

Our of curiosity, do you know of *any* client applications that actually 
support feed paging? Or have expressed an intention to support feed paging? 
By client application I mean something a user can download and install on 
their system and use as a feed reader. Last time I checked I couldn't find 
any on Windows. Is there better support on Linux and Macs?

Regards
James

---

From: "Thomas Broyer" <[email-masked]>
Date: Wed, 7 Jun 2006 15:02:06 +0200

[sorry, autocompletion got it to atom-protocol]

> As such, I'm pretty much convinced that they need to be dealt with

I'm not sure they're as different as you're saying…

From the client point of view, isn't an incremental feed just the same
as a paging feed, except that you're assured you won't loose entries
(well, that's not really true actually: "live feed document" can
change while you're following next/previous links, and could even
result in a new "archived feed document").

If applications need paging around snapshots to avoid loosing entries,
then they (servers) can create a snapshot of a query result set that
the client will page through (with some TTL before the snapshot is
deleted).

Actually, what changes is how clients deal with them, but those
different feeds probably won't be consume by the same clients: an
incremental feed is designed to be subscribed by a "feed reader" while
others are for more specific uses (search results, shared lists with
or without ranking info, etc.)
I'm not saying you won't use the same client application to process
those feeds, but the client is assumed to behave differently depending
whether it's dealing with an incremental feed or another paging feed
(i.e. a snapshot feed split into small "pages"). That behaviour is not
dependent upon the relations between the "chunks" but in how to deal
with the feed: if it's an incremental feed, I'll use a "feed state
reconstruction algorithm" to avoid downloading entries that have not
changed, and will probably also keep local copies of those entries; if
it's a "paged snapshot feed", the set of entries in the all the
"chunks" replaces the previous set, and some "algorithm"
(preconditions and/or IM) can still be used to try to avoid
downloading chunks that have not changed.

The only difference that *might* exist in dealing with the relations
is that, for an incremental feed, as soon as a requests returns a 304
(Not Modified), you can stop "paging", whereas for paged snapshot
feeds the next chunk might have changed…

But the real relationship between these pages/chunks/etc. is the same:
one comes after/before the other.
A "dumb" client could download the whole set of pages/chunks/etc. each
time, again and again, and will still have the same set of entries.
The "feed state reconstruction algorithm" is some kind of optimization
to help not wasting bandwidth: it doesn't change the way you follow
links, just the way you optimize retrievals.

So a "flag" in the feed telling which kind of optimization you can use
without loosing information is enough; something like the
fh:incremental element in version -04 of your draft.

> generic "previous" and "next".

I think I originally pushed the other way around [1], but for
different reasons. If it's clear that previous/next deals with "pages"
or "chunks" of the same feed, then I'm OK with them, and other use
cases will need new rel values.

[1] http://www.imc.org/atom-syntax/mail-archive/msg17372.html

> and friends) and paging feeds ("previous", "next" and friends).

Given the above, yes. Consider it a "-0" though, not a "-1", as I
might need some more thinking (or people trying to convince me).

--
Thomas Broyer

---

From: James M Snell <[email-masked]>
Date: Wed, 07 Jun 2006 07:45:46 -0700

We have a client that is specific to our Open Activities platform that
uses feed paging.  It's not a general feed reader, tho.  I believe our
general feed reader component will support paging (maybe does, haven't
checked on it in a while).

- James

James Holderness wrote:
> support feed paging? Or have expressed an intention to support feed
> paging? By client application I mean something a user can download and
> install on their system and use as a feed reader. Last time I checked I
> couldn't find any on Windows. Is there better support on Linux and Macs?
> 
> Regards
> James
> 
>

---

From: Mark Nottingham <[email-masked]>
Date: Wed, 7 Jun 2006 08:11:49 -0700

On 2006/06/07, at 4:26 AM, James Holderness wrote:

> I'm going to reserve judgement until I see what exactly it is  
> you're proposing, but I'm not particularly keen to change our  
> existing implementation.

Understood. I've been reluctant to change the spec for just that  
reason, but the split has become pretty apparent.

> Our of curiosity, do you know of *any* client applications that  
> actually support feed paging? Or have expressed an intention to  
> support feed paging? By client application I mean something a user  
> can download and install on their system and use as a feed reader.  
> Last time I checked I couldn't find any on Windows. Is there better  
> support on Linux and Macs?

I think most of the use cases for paging have to do with things like  
GData, OpenSearch, etc -- i.e., query results. That sort of thing  
isn't targetted at desktop aggregators AFAICT; it seems to be more  
for machine->machine communication, or for browsing a result set.

---

From: "James Holderness" <[email-masked]>
Date: Wed, 7 Jun 2006 17:03:26 +0100

Mark Nottingham wrote:
>> support feed paging?
> GData, OpenSearch, etc -- i.e., query results. That sort of thing  isn't 
> targetted at desktop aggregators AFAICT; it seems to be more  for 
> machine->machine communication, or for browsing a result set.

Actually when I said "feed paging" I meant that as a general term, i.e. feed 
history too.

As for machine->machine communication, if these feeds aren't meant for 
desktop aggregators then does it really matter that they function 
differently? You can describe one algorithm for use in machine->machine 
communication and another for use by desktop aggregators downloading 
"regular" feeds. Both can use the same link relations because they should 
never come into contact with each other. Having said that I still don't see 
how a machine->machine algorithm for retrieving a paged feed can be 
different from your current feed history algorithm and still be useful.

Lets say I was a search engine returning paged results. A search is 
performed that returns 200 results. I return 20 pages, 10 results per page. 
First time around a client supporting the feed history algorithm would 
retrieve all 20 pages no problem. So far I see no difference between how a 
desktop aggregator would behave and how machine->machine communication would 
function.

The second time the client connects (assuming there is a second time) it 
sends through an etag and/or last-modified date so the search engine knows 
which results it already has. Say there are 3 new results since the previous 
retrieval. Either the search engine is smart enough to just return those 3 
results or it's going to ignore the etag and return everything - 21 pages, 
10 results per page, new items could be anywhere.

As a desktop aggregator I guarantee you I'm not going to want to download 
20+ pages every hour just to find the 3 new items that *might* be there. 
Fortunately the feed history algorithm would stop me after the first page, 
and I'm thankful for that. Would a machine->machine communication be any 
different? Would they really want to download every single one of those 203 
results just to find the 3 new items?

If that's what they really want to do, then they can always ignore the "stop 
when you encounter a link you've already retrieved" part of the feed history 
algorithm. We don't need a special link relationship for that. If anything, 
I would think Thomas' suggestion of some kind of flag should be enough. I 
just can't see anyone wanting to do something like that, though, desktop 
aggregator or machine->machine communication.

Regards
James

---

From: Sylvain Hellegouarch <[email-masked]>
Date: Wed, 07 Jun 2006 18:15:46 +0100

> isn't targetted at desktop aggregators AFAICT; it seems to be more for 
Which sets the question of what is the target of the specification and 
Atom in general. We mainly center the discussion around aggregators but 
I hope you all agree that Atom aims at much more than simple news feed, 
right?

- Sylvain

---

From: John Panzer <[email-masked]>
Date: Wed, 07 Jun 2006 10:24:58 -0700

James Holderness wrote:

>>> actually support feed paging?
>>
>>
> Actually when I said "feed paging" I meant that as a general term, 
> i.e. feed history too.
>
> differently? You can describe one algorithm for use in 
> machine->machine communication and another for use by desktop 
> aggregators downloading "regular" feeds. Both can use the same link 
> relations because they should never come into contact with each other. 
> Having said that I still don't see how a machine->machine algorithm 
> for retrieving a paged feed can be different from your current feed 
> history algorithm and still be useful.

Our current "previous" links aren't permalinks; they're more like 
OpenSearch indexes:  http://example.org/feed.xml?start=50&len=10.  So 
clients can't usefully cache them the way they could, say, 
http://example.org/feed.xml?month=may2006.  We may well want to offer 
both types of links, even in the same feed, to support different 
traversals of the underlying resource(s).

One issue we have today is that we're compatible with neither OpenSearch 
nor feed-history :(. 

-John

---

From: Mark Nottingham <[email-masked]>
Date: Wed, 7 Jun 2006 10:33:34 -0700

On 2006/06/07, at 9:03 AM, James Holderness wrote:
>
> As for machine->machine communication, if these feeds aren't meant  
> for desktop aggregators then does it really matter that they  
> function differently? You can describe one algorithm for use in  
> relations because they should never come into contact with each  
> other. Having said that I still don't see how a machine->machine  
> algorithm for retrieving a paged feed can be different from your  
> current feed history algorithm and still be useful.

I don't see a clean split between machine-to-machine vs. desktop  
aggregator cases; for example, an incremental-style feed can be  
useful both on the desktop (to make sure I see all of your blog  
entries) as well as with processes (to make sure that my program  
doesn't miss a critical event if it has some downtime or loss of  
connectivity).

Similarly, some of the cases I've heard for paging-style feeds are  
with desktop clients (e.g., "get me the next results, please") and  
some are with processes (e.g., processing search results automatically).

The difference has more to do with a) what guarantees the server  
wants to provide, and b) what resources they're willing to devote  
towards meeting those guarantees.

> performed that returns 200 results. I return 20 pages, 10 results  
> per page. First time around a client supporting the feed history  
> algorithm would retrieve all 20 pages no problem. So far I see no  
> difference between how a desktop aggregator would behave and how  
> machine->machine communication would function.
>
> The second time the client connects (assuming there is a second  
> time) it sends through an etag and/or last-modified date so the  
> search engine knows which results it already has. Say there are 3  
> new results since the previous retrieval. Either the search engine  
> is smart enough to just return those 3 results or it's going to  
> ignore the etag and return everything - 21 pages, 10 results per  
> page, new items could be anywhere.
>
> As a desktop aggregator I guarantee you I'm not going to want to  
> download 20+ pages every hour just to find the 3 new items that  
> *might* be there. Fortunately the feed history algorithm would stop  
> me after the first page, and I'm thankful for that. Would a machine- 
> >machine communication be any different? Would they really want to  
> download every single one of those 203 results just to find the 3  
> new items?

These are pretty much the assumptions that I was making previously.  
The degree of precision that FH currently provides isn't desirable  
for search results. Feed History also requires that the server  
maintain state about a particular feed, which is unworkable for  
search results; e.g., to implement feed history for search results, a  
server would have to mint a whole new set of feed documents for every  
query, and keep them around. That's not workable for most search  
engines (Yahoo, Google, Amazon, whatever), so they need another  
option -- one that needs to be clearly distinct from FH.

This brings me to my other motivation -- I found that most people who  
use "previous" and "next" don't understand the assumptions that FH  
makes about archive stability, and point them at URIs like "http:// 
example.org/feed.atom?page=3". That will break the FH algorithm  
badly, reducing the value of the mechanism as a whole, because people  
will stop trusting it. The link relation for implementing the  
incremental approach needs to have the stability semantics baked in  
and explicit.

---

From: "M. David Peterson" <[email-masked]>
Date: Wed, 7 Jun 2006 11:54:00 -0600

You can use Atom for news feeds too?!

Wow!  So many uses, so little time (please know I am being
sarcastic...  Just agreeing with Sylvain's point in my own
"personalized style" sort of way ;)

-- 
<M:D/>

M. David Peterson
http://www.xsltblog.com/

---

From: "James Holderness" <[email-masked]>
Date: Wed, 7 Jun 2006 19:16:32 +0100

Mark Nottingham wrote:
> These are pretty much the assumptions that I was making previously.  The 
> degree of precision that FH currently provides isn't desirable  for search 
> results. Feed History also requires that the server  maintain state about 
> a particular feed, which is unworkable for  search results; e.g., to 
> implement feed history for search results, a  server would have to mint a 
> whole new set of feed documents for every  query, and keep them around.

Not necessarily. They only need to be able to sort results on a 
most-recently-discovered date. When a client connects to them with an etag 
representing date X, they return only those results have been discovered 
since date X. I can believe that there are search engines for which even 
that isn't feasible, but then FH as we know is not possible, and those feeds 
are essentially useless from a subscription point of view.

They can still use the paging links so you can connect to a search engine 
for a once-off paged set of results. They just need to return 304 on any 
subsequent requests (i.e. anything with an etag) in case someone makes the 
mistake of subscribing to one of those feeds. If you have something else in 
mind for that kind of server I'm curious to know what it is. In other words 
can you envision a server that wants to do paging, doesn't have enough state 
information to be able to do FH, but still would like to allow 
subscriptions? How would it work?

> This brings me to my other motivation -- I found that most people who  use 
> "previous" and "next" don't understand the assumptions that FH  makes 
> about archive stability, and point them at URIs like "http:// 
> example.org/feed.atom?page=3".

I don't see how changing the name of the link is going to magically fix 
that. If people aren't understanding the significance of archive stability, 
then perhaps it needs to be highlighted more prominently in the FH spec.

> That will break the FH algorithm badly

I don't think it's that bad. It loses some of its functionality, but it's no 
worse than having no paging at all. And you still have the ability to 
retrieve the full history on first download.

John Panzer wrote:
> http://example.org/feed.xml?month=may2006.

I understand that, but do you realise that those links are essentially 
useless for anything other than the first retrieval? As a client there's 
nothing I can do with them after the first download. I don't have a problem 
with that if that's all you can do, though. It's still better than no paging 
at all. And it still works (within the boundaries of what is feasible) using 
the FH algorithm as it currently stands. I don't see what else you can write 
into the spec that would make your feed work any better than it currently 
does.

Regards
James

---

From: "Thomas Broyer" <[email-masked]>
Date: Wed, 7 Jun 2006 20:30:56 +0200

> incremental approach needs to have the stability semantics baked in

As I previously said, the current FH algorithm isn't workable for all
the use cases (notably with paged non-incremental feeds, those feeds
being snapshots or "live feeds") but that doesn't mean there's a need
for other rel values.

I think FH could:
 - (more explicitly) RECOMMEND using stable "chunks" so that caching
can be used (maybe using an "overview" section similar to the APP's
section 5)
 - provide algorithms as an "how-to optimize bandwith et al. and not
retrieve the whole set of pages/chunks", not as a "you should/must do
that to comply to this spec"
 - provide different algorithms for incremental and non-incremental
feeds, non-incremental feeds retrieval having to go through all the
pages/chunks and not stop as soon as there's no more change (but the
whole algorithm remains almost the same)
 - RECOMMEND using RFC3229 w/ feeds (problem: this is not an I-D),
particularly for non-incremental feeds (Atom does not define order for
the entries in the feed, order has to be told by extensions, e.g. Feed
Rank, so there's no problem retrieving only a subset of the whole
feed, even if new/updated entries would be sparsed in the feed if
retrieved without RFC3229 w/ feeds), and NOT RECOMMEND (or even a
"MUST NOT") using paging for RFC3229 responses: either you return the
whole set of changes, or you choose to ignore the "IM" request and
return the feed as if the client didn't use RFC3229)
 - use 304 (Not Modified) instead of IRI-comparison, enlightening the
fact that web servers like Apache provide such response out-of-the-bow
when delivering files (this should incite people to using stable
chunks/pages by saving them to files, instead of making use of CGI
applications with database requests). The fact that many
implementations currently return feeds (whether or not they're using
paging/history) without processing preconditions (If-Modified-Since or
If-Not-Match) shouldn't influence the FH design, many of these
implementations are deployments of CMSes like WordPress, and those can
be fixed. Maybe the Feed Validator can have an option to test this: it
would retrieve the feed twice (at least), using If-Modified-Since
and/or If-Not-Match for the second request, and showing a warning if
the feed hasn't changed and the second GET didn't result in a 304 (Not
Modified).

Please note that I haven't went back read the FH draft since weeks, so
some of my comments might not be accurate (I've a pretty good memory
but data-losses might still happen ;-) )

-- 
Thomas Broyer

---

From: Mark Nottingham <[email-masked]>
Date: Wed, 7 Jun 2006 14:33:02 -0700

On 2006/06/07, at 11:16 AM, James Holderness wrote:

> Mark Nottingham wrote:
>> These are pretty much the assumptions that I was making  
>> previously.  The degree of precision that FH currently provides  
>> isn't desirable  for search results. Feed History also requires  
>> that the server  maintain state about a particular feed, which is  
>> unworkable for  search results; e.g., to implement feed history  
>> for search results, a  server would have to mint a whole new set  
>> of feed documents for every  query, and keep them around.
>
> most-recently-discovered date. When a client connects to them with  
> an etag representing date X, they return only those results have  
> been discovered since date X. I can believe that there are search  
> engines for which even that isn't feasible, but then FH as we know  
> is not possible, and those feeds are essentially useless from a  
> subscription point of view.
>
> They can still use the paging links so you can connect to a search  
> engine for a once-off paged set of results. They just need to  
> return 304 on any subsequent requests (i.e. anything with an etag)  
> in case someone makes the mistake of subscribing to one of those  
> feeds. If you have something else in mind for that kind of server  
> I'm curious to know what it is. In other words can you envision a  
> server that wants to do paging, doesn't have enough state  

I'm not sure how ETags and 304s come into it -- it sounds like you're  
proposing using either the entry-level updated date or the entry- 
level id as input to a server-side function to select a set of  
entries from the feed.  Can you paint out your proposal in protocol  
exchanges, please?

---

From: "James Holderness" <[email-masked]>
Date: Wed, 7 Jun 2006 23:35:26 +0100

Mark Nottingham wrote:
> proposing using either the entry-level updated date or the entry- level id 
> as input to a server-side function to select a set of  entries from the 
> feed.  Can you paint out your proposal in protocol  exchanges, please?

Entry-level updated date is close, but not quite what would be needed. For 
this to work, it requires that search engines store a date that represents 
when a result is "discovered". In other words the date that an entry is 
added to the search engine database.

Client connects to server with a specific query.
Server returns the first page of results along with an etag representing the 
current internal datetime as accurately as possible.
Client connects to server with "next" link from first page (the link would 
obviously have to include the query as well as the page number).
Server returns the second page of results.
etc.

<wait several days>

Client connects to server with a specific query along with etag returned 
from the first query.
Server returns only those results that match the query *and* have a 
"discovered" date >= etag date value.
Server also returns etag representing the current internal datetime as 
before.
Client connects to server with "next" link from first page (this link would 
include the query and the page number, but also the the etag value).
Server returns the second page of results that match the query *and* have a 
"discovered" date >= etag value.
etc.

That make sense?

If the server doesn't have the concept of a "discovered" date, then there's 
not much we can do. We can return a paged set of results, but they can't be 
updated in any meaningful way so the feed should not be subscribed to.

Client connects to server with a specific query.
Server returns first page of results along with an etag containing a hash of 
the star-spanngled banner.
Client connects to server with "next" link from first page.
Server returns the second page of results.
etc.

<wait several days>

Client connects to server with a specific query along with etag returned 
from initial query.
Server notices that there is an etag, doesn't care what it is set to, and 
just returns 304 regardless.

In other words, you can retrieve the feed once, but never again. That's as 
good as it gets.

Regards
James

---

From: Mark Nottingham <[email-masked]>
Date: Wed, 7 Jun 2006 17:06:40 -0700

Are you talking about using ETag HTTP response headers, If-Match  
request headers, and 304 Not Modified response status codes? That's a  
gross misapplication of those mechanisms if so, and this will break  
intermediaries along the path.

Even if it's cast as a query parameter in the URI (for example), it  
requires query support on the server side, a concept of "discovered  
time" (as you point out), and places constraints on the ordering of  
the feed.

Are you proposing this instead of the mechanism currently described  
in FH? Alongside it?

On 2006/06/07, at 3:35 PM, James Holderness wrote:

>
> Mark Nottingham wrote:
>> I'm not sure how ETags and 304s come into it -- it sounds like  
>> you're proposing using either the entry-level updated date or the  
>> entry- level id as input to a server-side function to select a set  
>> of  entries from the feed.  Can you paint out your proposal in  
>> protocol  exchanges, please?
>
> Entry-level updated date is close, but not quite what would be  
> needed. For this to work, it requires that search engines store a  
> date that represents when a result is "discovered". In other words  
> the date that an entry is added to the search engine database.
>
> Server returns the first page of results along with an etag  
> representing the current internal datetime as accurately as possible.
> Client connects to server with "next" link from first page (the  
> link would obviously have to include the query as well as the page  
> number).
> Client connects to server with a specific query along with etag  
> returned from the first query.
> Server also returns etag representing the current internal datetime  
> as before.
> Client connects to server with "next" link from first page (this  
> link would include the query and the page number, but also the the  
> etag value).
> Server returns the second page of results that match the query  
> *and* have a "discovered" date >= etag value.
> etc.
>
> That make sense?
>
> If the server doesn't have the concept of a "discovered" date, then  
> there's not much we can do. We can return a paged set of results,  
> but they can't be updated in any meaningful way so the feed should  
> not be subscribed to.
>
> Server returns first page of results along with an etag containing  
> a hash of the star-spanngled banner.
> Client connects to server with a specific query along with etag  
> returned from initial query.
> Server notices that there is an etag, doesn't care what it is set  
> to, and just returns 304 regardless.
>
> In other words, you can retrieve the feed once, but never again.  
> That's as good as it gets.
>
> Regards
> James
>

---

From: "James Holderness" <[email-masked]>
Date: Thu, 8 Jun 2006 01:43:17 +0100

Mark Nottingham wrote:
> Are you talking about using ETag HTTP response headers, If-Match  request 
> headers, and 304 Not Modified response status codes? That's a  gross 
> misapplication of those mechanisms if so, and this will break 

For the first page I'm talking about an Etag (or Last-Modified) HTTP 
response header and If-None-Match (or If-Modified-Since) request headers for 
the retrievals a month later. For page two onwards the state information 
(date, query and page number) comes from the link urls returned by the first 
page. I don't see how that's a gross misapplication of those mechanisms, but 
I'll take your word for it.

> requires query support on the server side, a concept of "discovered  time" 
> (as you point out), and places constraints on the ordering of  the feed.

The ordering is not necessarily important. As long as the server can filter 
out entries that don't match a specific time criteria it can return those 
entries in any order.

> Are you proposing this instead of the mechanism currently described  in 
> FH? Alongside it?

What I'm proposing would work with the FH as currently specified as long as 
the client supported ETag or Last-Modified as well. For me that means no 
change at all.

And I'm not trying to propose this as something that servers have to do. It 
just seemed to me to be a fairly easy way for a search engine to support 
subscribable paging if they chose to. And it should work with clients that 
already support FH so it doesn't require any new links or algorithms.

If you have something better in mind that's cool too. So far the plan seems 
to be:

1. Change the names of the link relations so as to break existing clients.
2. ???
3. Profit!!

Regards
James

---

From: "Thomas Broyer" <[email-masked]>
Date: Thu, 8 Jun 2006 08:40:18 +0200

2006/6/8, James Holderness <[email-masked]>:
> the retrievals a month later.

What you described is RFC3229 w/ feeds [1], but you failed to include
the new request and response headers and the specific status code,
which are necessary because you're changing the behaviour of
If-None-Match and 304 (Not Modified) as defined in HTTP/1.1.

> For page two onwards the state information (date, query and page number)
> comes from the link urls returned by the first page.

That means you need to keep entry revisions as well, so that if an
entry is updated while a client is navigating the paged result set, it
is sent the "old" revision (corresponding to the "date" parameter).

> out entries that don't match a specific time criteria it can return those

Yes, ordering is not important. If ranking is necessary, then use the
Feed Rank extension (but that means that potentially a great number of
entries will be sent back as "modified" in 226 (IM Used) responses
just because their ranking has changed)

> the client supported ETag or Last-Modified as well. For me that means no

You're trying to change HTTP/1.1 behaviour wrt the If-None-Match
request-header field and the 304 (Not Modified) status code, so you
need to implement RFC3229 w/ feeds (which means dealing with some new
headers and a new status code).

As I already said, I highly suggest not using paging for 226 (IM Used)
responses and rather fall back to "standard GET" in case there are too
many changes (i.e. behaving the same way as servers that don't support
RFC3229 w/ feeds).

My main concern is that RFC3229 w/ feeds is being deployed more and
more widely and is still not even an I-D (or I missed something).
Maybe FH could be the place to spec it, as another optimization
algorithm…

[1] http://bobwyman.pubsub.com/main/2004/09/using_rfc3229_w.html

-- 
Thomas Broyer

---

From: "James Holderness" <[email-masked]>
Date: Thu, 8 Jun 2006 10:37:28 +0100

> If-None-Match and 304 (Not Modified) as defined in HTTP/1.1.

Yep. Sorry, forgot to mention that.

> is sent the "old" revision (corresponding to the "date" parameter).

Why? If an entry has been revised either don't send it (they'll get it then 
next time they refresh), or send it anyway (they'll just get it again the 
next time they refresh).
Is that such a big deal? Or am I missing something?

> just because their ranking has changed)

I would have thought IM only applied to the first page. All subsequent pages 
have a specific query that includes the query, page and time. You're not 
sending back a partial result in that case.

> headers and a new status code).

No change at all for *me*. As in my client. I already support FH. I already 
support Etags. I already support 3229.

> many changes (i.e. behaving the same way as servers that don't support

I don't get why this is a problem, but if you don't like it don't use it.

All I'm saying is, if you're a search engine and you what to create 
subscribable paged results, this is a method that you can use right now, and 
it will work with at least one existing FH capable client (I suspect others 
too). The other proposal on the table is to change all your link names. 
Arguably a much better proposal than what I'm offering - it certainly seems 
to have got a lot of +1s - but it will work with precisely no one.

Regards
James

---

From: "Thomas Broyer" <[email-masked]>
Date: Thu, 8 Jun 2006 12:05:59 +0200

> Is that such a big deal? Or am I missing something?

Sorry, I thought you wanted search engines to produce "snapshots"...

(side note: but in this case, is there a need to pass a "date"
parameter to following pages? and if pages are kind of "live", isn't
there a risk of data loss? –I mean, this is the Web, so you'll end up
doing the request for each page, just returning different chunks of
the result set; if an entry changes between the request to the first
page and the retrieval a following page, your request might put it
somewhere else in the result set, changing ordering of entries based
on updated time stamps, "discovery date", ranks or else, so your
chunks would be different than if the entry hadn't changed, and an
entry that have not been retrieved might end up in an already
retrieved "chunk by page number", hence the client missing an entry– I
think this is Mark's concern: this might be an acceptable behaviour in
some cases but not all)

> support Etags. I already support 3229.

OK, I though you only supported Etags, as defined by HTTP/1.1 for
efficient caching and bandwidth saving.

> I don't get why this is a problem, but if you don't like it don't use it.

Yep, sorry, this is not a problem.

> too).

So we agree ;-)

Could you read my recent mails in this thread and confirm that it's the case?

> The other proposal on the table is to change all your link names.

So there now are two -1, isn't it? ;-)

-- 
Thomas Broyer

---

From: Mark Nottingham <[email-masked]>
Date: Thu, 8 Jun 2006 11:40:07 -0700

> more widely and is still not even an I-D (or I missed something).

I have that concern as well.

I am also concerned that RFC3229 is an extension of HTTP, but some  
implementers are acting as if it chages the semantics of already- 
defined parts of HTTP. For example, a delta must be a subset of the  
current representation that is returned to a GET; if you GET the  
feed, it has to return all of the entries that you could retrieve by  
using delta.

I have a feeling that many people are treating it as a dynamic query  
mechanism that's capable of retrieving any entry that's ever been in  
the feed, while still only returning the last n entries to a plain  
GET. If so, they're breaking HTTP, breaking delta, and should use  
something else.

Is this the case, or am I (happily) mistaken?

---

From: "James Holderness" <[email-masked]>
Date: Thu, 8 Jun 2006 21:45:09 +0100

> Could you read my recent mails in this thread and confirm that it's the 

I'm sorry, but I can no longer participate in this discussion. I hope 
everything works out ok.

Regards
James

---

From: "Thomas Broyer" <[email-masked]>
Date: Fri, 9 Jun 2006 09:36:43 +0200

> Is this the case, or am I (happily) mistaken?

RFC3229 adds a new notion, the one of "instance" and defines that the
ETag is computed based on the instance, not the entity (difference
between instance and entity being the instance-manipulation –delta,
diff, or identity, which is the default in bare HTTP– having been
applied).

Now, let's define the "feed resource" as the complete set of entries,
and its default representation as one which shows only the most recent
ones (using a yet-to-be-defined conneg algorithm –e.g. using an
Accept-Features feature–, it could be possible to retrieve the whole
set of entries).

If I'm wront in the above statement (conneg could lead to a "latest
entries only" or "full", without abusing HTTP), then don't even read
further, this would mean that RFC3229 w/ feed changes the semantics of
existing parts of HTTP.

Let's take the steps in section 4 of RFC3229:
>   Conceptually, the various transformations are applied in the
>   following sequence:
>
>      1. Upon receiving a GET request, the server uses the URI in the
>         request to identify the requested resource.
>
>      2. Optionally, it uses information from the request (and perhaps
>         additional information) to select a variant of that resource.

The server might here use the presence of the A-IM request header to
select the "full" representation of the feed rather than the "latest
entries only" representations.

>      3. At this point, the server may apply a non-identity content-
>         coding to the instance, or one might have been inherent in its
>         generation.  This also results in a Content-Encoding header.
>
>      4. The result of the first three steps, at the time when the
>         request is processed, is an instance.  The instance includes a
>         body (possibly empty) and possibly some instance headers.  The
>         entity tag, if any, is assigned at this point.  That is, an
>         entity tag is associated with an instance, NOT an entity.

If I read RFC2616 correctly, nothing prevents the "latest entries
only" and "full" representations (or any representation of the same
resource) to share the same entity-tag, i.e. nothing prevents a server
to assign an entity-tag to a resource rather than its representations.
Maybe servers should use a weak etag in this case? But RFC2616 says in
section 13.3.3 that "Clients MAY issue simple (non-subrange) GET
requests with either weak validators or strong validators. Clients
MUST NOT use weak validators in other forms of request."

>      5. The server may then apply an instance-manipulation.  For
>         example, if the request included a Range header, the server may
>         optionally produce a range response, consisting of the original
>         set of headers, a Content-Range header, and the appropriate
>         range(s) from the (possibly encoded) body.  Delta encodings are
>         instance-manipulations, and are computed at this stage.

Given the ETag of the previous "version" of the feed, the server can
compute the delta and make it a feed of added and updated entries.

>      6. The result of the fifth step becomes the entity, consisting of
>         entity headers and an entity body.
>
>      7. The server may then apply a non-identity transfer-coding; on-
>         the-fly compression could be done in this step.  If so, a
>         Transfer-Encoding header is added to the message.
>
>      8. The results of the seventh step is the message, consisting of a
>         message body (the transfer-coded version of the entity body),
>         the entity headers, and additional response and general
>         headers.

-- 
Thomas Broyer

---

From: Julian Reschke <[email-masked]>
Date: Fri, 09 Jun 2006 10:14:31 +0200

Thomas Broyer schrieb:
> ...

I think this is a misunderstanding of how HTTP/1.1 is supposed to work.

If a server returns the same ETag for different entities returned for 
the same URL, this will cause breakage of caches. You may want to 
re-read Section 13.6 of RFC2616 
(<http://greenbytes.de/tech/webdav/rfc2616.html#rfc.section.13.6>).

Best regards, Julian

---

From: "Thomas Broyer" <[email-masked]>
Date: Fri, 9 Jun 2006 11:22:04 +0200

> (<http://greenbytes.de/tech/webdav/rfc2616.html#rfc.section.13.6>).

But using A-IM, you'll have Cache-Control and Vary headers in
responses, which should take care of that, no?
(I'm at work have no time to read the pointed section carefully, so
please forgive me if I'm wrong)

-- 
Thomas Broyer