Subject: Re: Feed update mechanism

From: David Powell <[email-masked]>
Date: Tue, 16 May 2006 12:09:31 +0100

Tuesday, May 16, 2006, 11:18:04 AM, Sylvain Hellegouarch wrote:

> Hi everyone,

> These days It seems that when UAs request a server to check if a feed has
> changed the server responds with either an HTTP 304 Not Modified status
> code or by returning the updated feed.

> It looks to me as a problem if only one or a couple of entries have been
> updated within the feed as it wastes so much bandwith specially when the
> said feed contains all entries being produced on the server.

Check out "A-IM: feed" [1]. It is already implemented by lots of
aggregators, and it is trivial for client implementors to support.

[1] http://bobwyman.pubsub.com/main/2004/09/using_rfc3229_w.html

-- 
Dave

---

From: "Sylvain Hellegouarch" <[email-masked]>
Date: Tue, 16 May 2006 12:55:47 +0100 (BST)

Hi Dave,

> [1] http://bobwyman.pubsub.com/main/2004/09/using_rfc3229_w.html

Very nice article indeed and it shows I'm not the only wondering about
this. The A-IM idea is pretty neat and efficient apparently. I'll give it
a look.

Thanks,
- Sylvain

---

From: "Sylvain Hellegouarch" <[email-masked]>
Date: Tue, 16 May 2006 11:18:04 +0100 (BST)

Hi everyone,

Below are ideas I had about that issue. Nothing formal.

PROPOSAL 1
==========

1. An UA requests the feed for the first time, the server returns the full
current feed. The UA stores it in its cache
2. The UA requests the feed again and provide the If-Modified-Since header
   a. No changes done, the server returns a 304 Not Modified
   b. The server checks which entries have been updated or published since
that date and returns a feed containing only those. The UA updates its
copy of the feed with the incoming data.

The big issue with this algorithm is that it adds semantic to the HTTP
caching system by expecting the UA to update its copy of the feed instead
of simply replacing it. Although specific aggregators could do it, it is
more than likely that some clients would just replace their copy of the
feed and confuse the end user.

PROPOSAL 2
==========

1. An UA requests the feed for the first time, the server returns the full
current feed. The UA stores it in its cache
2. The UA requests the feed again but provides a time range like this:
http://host/feed?after=2003-12-13T18:30:02Z&before=2003-12-13T19:30:02Z
   a. No changes done, the server returns a 304 Not Modified
   b. If entries have been published or updated in the time range, then
return a feed with those only.

Of course, one might only pass either 'after' or 'before'.
The problem with this proposal is to add parameters to the URI which might
not be handled by the server. There are a few ways to handle that:
   * The UA could also use the OPTIONS method from HTTP to check if the
server supports the said URI.
   * The client sends the requests and the server could return 400 Bad
Request if it cannot handle it.

In order to deal with those proposals we could add a simple extension to a
feed such as <atom:feed update-method="full|incremental|chunk">

When receiving the feed for the first time the client would know if it can
either request for:
     * full => default to fetch the complete feed each time when it has
been modified. The default behavior as it exists today.
     * incremental => the server says it can serve entries published or
updated after a given date (equivalent to using only the 'after'
parameter in PROPOSAL 2 or PROPOSAL 1)
     * chunk => the server says it can serve feed of entries published or
updated between two dates (as explained in PROPOSAL 2)

These are just some thoughts and there might already be ways of dealing
with this. If not I'd be happy to hearing your feedback.

- Sylvain
http://www.defuze.org

---

From: "M. David Peterson" <[email-masked]>
Date: Tue, 16 May 2006 20:32:56 -0600

Worth noting:

via
http://www.oreillynet.com/windows/blog/2006/04/keep_the_good_things_rolling_m.htmlwhich
links to
http://blogs.msdn.com/rssteam/archive/2006/04/08/571509.aspx you'll find:

Delta Encoding

In addition to standard conditional GETs the RSS Platform download engine
supports Delta Encoding (for details, see Bob Wyman's post "Using RFC 3229
with Feeds") which allows the server to respond with only the feed items
that are new or have been updated, thereby possibly reducing the response
significantly.

So RFC3229 is supported by default in IE7... Good stuff for sure :)

-- 
<M:D/>

M. David Peterson
http://www.xsltblog.com/

---

From: "Sylvain Hellegouarch" <[email-masked]>
Date: Wed, 17 May 2006 10:00:36 +0100 (BST)

> http://blogs.msdn.com/rssteam/archive/2006/04/08/571509.aspx you'll find:

Very nice article David, I really like the safe approach taken by
Microsoft on this one.

Thanks,
- Sylvain