Subject: Reader 'updated' semantics

From: "James Yenne" <[email-masked]>
Date: Mon, 9 Jan 2006 16:29:45 -0800

I'm looking for info about what the expected 'updated' datetime element
semantics look like, and haven't found anything in old email and the wiki on
this topic. Is there a description somewhere?  RSSBandit and FeedDemon don't
pick up entry changes based on my changing the feed and entry 'updated'
element to a later datetime.  Instead I change the id element and get a
'new' entry, which is not desirable, since id for an entry should be a
permanent uuid.  I don't know how updates are supposed to occur.
 
Thanks for any help,
James

---

From: James M Snell <[email-masked]>
Date: Mon, 09 Jan 2006 17:08:10 -0800

The updated element is used to indicate when a significant update has 
occurred to the entry.  If you are updating the updated element when you 
update your entry, you are doing the right thing.  If RSSBandit and 
FeedDemon are not picking up the fact that the entry has been modified 
based on a changed updated value (even if the updated element is the 
only thing that is different), then I would say those readers have a 
bug.  You should never be changing your id element value.

James Yenne wrote:
> semantics look like, and haven't found anything in old email and the 
> wiki on this topic. Is there a description somewhere?  RSSBandit and 
> FeedDemon don't pick up entry changes based on my changing the feed and 
> entry 'updated' element to a later datetime.  Instead I change the id 
> element and get a 'new' entry, which is not desirable, since id for an 
> entry should be a permanent uuid.  I don't know how updates are supposed 
> to occur.
>  
> Thanks for any help,
> James
>  
>

---

From: Tim Bray <[email-masked]>
Date: Mon, 09 Jan 2006 21:30:44 -0800

On Jan 9, 2006, at 5:08 PM, James M Snell wrote:

> The updated element is used to indicate when a significant update  
> has occurred to the entry.  If you are updating the updated element  
> when you update your entry, you are doing the right thing.  If  
> RSSBandit and FeedDemon are not picking up the fact that the entry  
> has been modified based on a changed updated value (even if the  
> updated element is the only thing that is different), then I would  
> say those readers have a bug.  You should never be changing your id  
> element value.

James is exactly right.  The whole reason 'updated' exists is  
*exactly* so you can tell the downstream software unambiguously when  
some entry has changed and this ought to be brought to the consumers'  
attention.  -Tim

---

From: "James Yenne" <[email-masked]>
Date: Tue, 10 Jan 2006 07:06:59 -0800

According to an exchange on the topic at RSSBandit's support site between
Dare and myself, he writes: 

"RSS Bandit does not provide an indication of an "updated" entry since many
differ on what constitutes an update and what level of feedback should be
provided to the user." 

I'm led to believe there is some controversy here.  James' description is
what I would expect, and seems straight-forward enough.  Is there any thing
more to this?  

Thanks 

-----Original Message-----
From: [email-masked] [mailto:[email-masked]]
On Behalf Of Tim Bray
Sent: Monday, January 09, 2006 9:31 PM
To: James M Snell
Cc: James Yenne; [email-masked]
Subject: Re: Reader 'updated' semantics

On Jan 9, 2006, at 5:08 PM, James M Snell wrote:

> occurred to the entry.  If you are updating the updated element when 
> you update your entry, you are doing the right thing.  If RSSBandit 
> and FeedDemon are not picking up the fact that the entry has been 
> modified based on a changed updated value (even if the updated element 
> is the only thing that is different), then I would say those readers 
> have a bug.  You should never be changing your id element value.

James is exactly right.  The whole reason 'updated' exists is
*exactly* so you can tell the downstream software unambiguously when some
entry has changed and this ought to be brought to the consumers'  
attention.  -Tim

---

From: Stephane Bortzmeyer <[email-masked]>
Date: Tue, 10 Jan 2006 16:35:01 +0100

On Tue, Jan 10, 2006 at 07:06:59AM -0800,
 James Yenne <[email-masked]> wrote 
 a message of 39 lines which said:

> "RSS Bandit does not provide an indication of an "updated" entry
> since many differ on what constitutes an update and what level of
> feedback should be provided to the user."

Sage displays entries sorted according to <updated> but does not
highlight updated entries, only new ones (new == new <id>). I believe
it is partly an User Interface and Usability issue and therefore out
of scope for the Atom-syntax group.

---

From: "James Yenne" <[email-masked]>
Date: Tue, 10 Jan 2006 08:12:45 -0800

I haven't used Sage, but I would anticipate that entries which have been
retrieved initially by a reader are continually checked for changes to the
updated element until the entries age to some drop off point, regardless of
the UI display conventions of them.  The problem is that there is data loss
in the reader that doesn't update entries that have actually changed, again,
regardless of how you display these changes.  Perhaps semantics is indeed
outside of the scope of a syntax discussion list, however, the updated
element syntax implies some semantic understanding.  I am not a reader
implementor, so I don't have any insight on why readers do not pick up these
changed updated elements as a cue for entry refresh.  Otherwise, please
direct me to a group that discusses these issues.  

Thanks

-----Original Message-----
From: [email-masked] [mailto:[email-masked]]
On Behalf Of Stephane Bortzmeyer
Sent: Tuesday, January 10, 2006 7:35 AM
To: James Yenne
Cc: [email-masked]
Subject: Re: Reader 'updated' semantics

On Tue, Jan 10, 2006 at 07:06:59AM -0800,  James Yenne
<[email-masked]> wrote  a message of 39 lines which said:

> "RSS Bandit does not provide an indication of an "updated" entry since 
> many differ on what constitutes an update and what level of feedback 
> should be provided to the user."

Sage displays entries sorted according to <updated> but does not highlight
updated entries, only new ones (new == new <id>). I believe it is partly an
User Interface and Usability issue and therefore out of scope for the
Atom-syntax group.

---

From: "A. Pagaltzis" <[email-masked]>
Date: Tue, 10 Jan 2006 17:32:08 +0100

Hi James,

* James Yenne <[email-masked]> [2006-01-10 16:20]:
>According to an exchange on the topic at RSSBandit's support
>site between Dare and myself, he writes: 
>
>since many differ on what constitutes an update and what level
>of feedback should be provided to the user." 

The reader I use, Liferea, specifically marks entries as edited
when they change, but not as unread. I’d prefer if it also gave
indication of the number of edited entries in a feed much as it
gives for the number of unread entries, but much to my chargin,
the developers have chosen not to support any indication for
edited entries beyond marking them as such.

It’s certainly something I plan to support once I get around to
writing the aggregator I’ve been sketching out in my head.

>I'm led to believe there is some controversy here.  James'
>description is what I would expect, and seems straight-forward
>enough. Is there any thing more to this?  

I don’t think it’s controversy, so much as that most people
apparently simply don’t care whether an entry they’ve already
seen has changed. I do, but I seem to be in a minority, since few
aggregators make this easy, and noone complains, whereas,
according to the Liferea crew, there have been complaints from
people who do *not* care when edited entries *were* called out.

It’s not a spec issue so much as a usability question.

Regards,
-- 
Aristotle Pagaltzis // <http://plasmasturm.org/>

---

From: "James Holderness" <[email-masked]>
Date: Tue, 10 Jan 2006 16:58:12 -0000

> what I would expect, and seems straight-forward enough.  Is there any 

There are a couple of options for an aggregator author. They can mark an 
entry as having changed when 1) the content of the entry has changed; 2) the 
updated element has changed; 3) the updated element has changed as well as 
the content having been changed; or 4) do nothing at all. I'm sure there are 
other possibilities, but I think those are the main issues.

With option 1 the poor user is flooding with notifications for all sorts of 
insignificant updates like single character spelling corrections. With 
option 2 the user is even worse off because they get notified of changes to 
an entry when in fact nothing at all in the content has changed (this is not 
uncommon). Option 3 is probably the least annoying, but can still end up 
flagging insignificant updates and can still miss significant updates if the 
publisher forgets to set the updated element. Option 4, well, it's easy if 
nothing else.

FWIW Snarfer does something based on option 3. Only changes to the actual 
message content are considered important enough to notify the user of an 
update. The title, author and other metadata aren't considered significant. 
Changes to the HTML markup in the content aren't considered significant 
either. Not perfect perhaps, but I think it strikes a fair balance between 
being useful and being annoying.

Regards
James

---

From: "James Yenne" <[email-masked]>
Date: Tue, 10 Jan 2006 08:55:47 -0800

As an example of the problem, if I'm using a reader as the sole means to get
my news, I would have missed the update to the coal miners story, from
miracle to tragedy, which was a very significant update.

-----Original Message-----
From: [email-masked] [mailto:[email-masked]]
On Behalf Of James Yenne
Sent: Tuesday, January 10, 2006 8:13 AM
To: 'Stephane Bortzmeyer'
Cc: [email-masked]
Subject: RE: Reader 'updated' semantics

Thanks

---

From: James M Snell <[email-masked]>
Date: Tue, 10 Jan 2006 09:07:06 -0800

In RSS there is definite confusion on what constitutes an update. In
Atom it is very clear.  If <updated> changes, the item has been updated.
     No controversy at all.

---

From: Tim Bray <[email-masked]>
Date: Tue, 10 Jan 2006 10:35:27 -0800

> Atom it is very clear.  If <updated> changes, the item has been  

Indeed.  There's a word for behavior of RssBandit and Sage: WRONG.   
Atom provides a 100% unambiguous way to say "this is the same entry,  
but it's been changed and the publisher thinks the change is  
significant."  Software that chooses to hide this fact from users is  
broken - arguably dangerous to those who depend on receiving timely  
and accurate information - and should not be used. -Tim

---

From: Stephane Bortzmeyer <[email-masked]>
Date: Wed, 11 Jan 2006 09:21:23 +0100

On Tue, Jan 10, 2006 at 10:35:27AM -0800,
 Tim Bray <[email-masked]> wrote 
 a message of 14 lines which said:

> There's a word for behavior of RssBandit and Sage: WRONG.   

Read again my message. Sage does *not* ignore changes of
<updated>. Its behaviour is perfectly right. It just displays updated
entries in a more discreet way than new entries.

> Software that chooses to hide this fact from users is broken

Sage does not hide it. Test before criticizing.

---

From: Stephane Bortzmeyer <[email-masked]>
Date: Wed, 11 Jan 2006 09:19:44 +0100

On Tue, Jan 10, 2006 at 04:58:12PM -0000,
 James Holderness <[email-masked]> wrote 
 a message of 31 lines which said:

> There are a couple of options for an aggregator author. They can
> mark an entry as having changed when 1) the content of the entry has
> changed; 2) the updated element has changed; 3) the updated element
> has changed as well as the content having been changed; or 4) do
> nothing at all. I'm sure there are other possibilities, but I think
> those are the main issues.

This is, IMHO, a completely different issue from the one asked by the
OP. In Atom, it seems to me that 2) is the only reasonable choice (1
or 3 would require to store the content - or at least a hash - and, if
applied blindly, would create many false positives since a simple
reformatting of the XML would trigger a "change").

The question asked by the OP was: when <updated> was changed, what the
reader should do? (And my reply was "Interesting question but a bit
out of scope for the atom-syntax group.")

---

From: Stephane Bortzmeyer <[email-masked]>
Date: Wed, 11 Jan 2006 09:16:47 +0100

On Tue, Jan 10, 2006 at 05:32:08PM +0100,
 A. Pagaltzis <[email-masked]> wrote 
 a message of 37 lines which said:

> I don???t think it???s controversy, so much as that most people
> apparently simply don???t care whether an entry they???ve already
> seen has changed.

It is also may be because feed writers are too eager to set <updated>
when the change is insignificant (for instance, when there is simply a
reformatting). In Atom, the specification is, IMHO, very clear:

RFC 4287, 4.2.15.  The "atom:updated" Element

   The "atom:updated" element is a Date construct indicating the most
   recent instant in time when an entry or feed was modified in a way
   the publisher considers significant.  Therefore, not all
                 ^^^^^^^^^^^^^^^^^^^^^
   modifications necessarily result in a changed atom:updated value.

But it is much more muddy in the RSS world which may explain why feed
reader programmers tend to ignore <updated>.

Developing best practices in this area is complicated because there is
a strong interaction between the feed writers and the reader software.

PS: Wikipedia allow authors to check a box "Minor update" when they
modify a page and they don't want it to be regarded as a significant
change. I wonder how many authors set it. It would be a nice Usability
study to examine that.

---

From: Bill de hÓra <[email-masked]>
Date: Wed, 11 Jan 2006 09:34:22 +0000

Tim Bray wrote:
> broken - arguably dangerous to those who depend on receiving timely  and
> accurate information - and should not be used. -Tim

I agree with this, up to the point that flipping the updated bit becomes
a spam vector.

cheers
Bill

---

From: "James Holderness" <[email-masked]>
Date: Wed, 11 Jan 2006 19:01:48 -0000

> reformatting of the XML would trigger a "change").

Read my response again. Option 3 can't possibly trigger any more false 
positives than 2 since it requires option 2's conditions to be satisfied 
(i.e. the updated element changed) in addition to the content having been 
changed.

Regards
James

---

From: "James Yenne" <[email-masked]>
Date: Thu, 12 Jan 2006 09:33:32 -0800

Thanks for your thoughtful analysis on the update issue.

I just want to follow up with reader implementers the importance of
supporting the Atom update mechanism.  News is one large application of a
feed, but there are potentially many others in which the update mechanism is
a requirement.

For feeds, with respect to email, this is clearly one differentiating factor
in which delivery is possible without creating a new message/entry.  At this
point, most Atom readers do not support <updated>, so the feed generator is
stuck with new = new id.

-----Original Message-----
From: [email-masked] [mailto:[email-masked]]
On Behalf Of James Holderness
Sent: Wednesday, January 11, 2006 11:02 AM
To: [email-masked]
Subject: Re: Reader 'updated' semantics

Regards
James