Closed Bug 1110273 Opened 10 years ago Closed 8 years ago

Old posts from hacks.mozilla.org repeatedly appear as new

Categories

(Websites :: planet.mozilla.org, defect)

Product:
Websites
Component:
planet.mozilla.org
Type:
defect
Priority:
Not set
Severity:
normal

Tracking

(Not tracked)

Status:
RESOLVED FIXED

People

(Reporter: andershol, Assigned: mhoye)

References

Details

It seems that periodically (perhaps every time a new post is added to the hacks.mozilla.org blog) new copies of the latest ~5 posts from the hacks.mozilla.org appear as new in my feed reader (thunderbird) for the planet feed. This is a overview of how many duplicates of the post have showed up from the first that showed up duplicated:

- "interact.js for drag and drop, resizing and multi-touch gestures" - 12 Nov 2014 - 3 times
- "Visually Representing Angular Applications" - 13 Nov 2014 - 1 time
- "Creating a mobile app from a simple HTML site - part 2." - 26 Nov 2014 - 7 times
- "Save the Web – Be a Ford-Mozilla Open Web Fellow" - 26 Nov 2014  - 1 time
- "MetricsGraphics.js – a lightweight graphics library based on D3" - 02 Dec 2014 - 5 times
- "Firebug 3 & Multiprocess Firefox (e10s)" - 03 Dec 2014 - 4 times
- "Mozilla Hacks gets a new Editor" - 04 Dec 2014 - 4 times
- "Videos and Firefox OS" - 05 Dec 2014 - 4 times
- "QuaggaJS – Building a barcode-scanner for the Web" - 10 Dec 2014 - 1 time

When the latest post appeared today posts back to "MetricsGraphics.js – a lightweight graphics library based on D3" showed up as new.
Looking into it, thanks for reporting this.
Assignee: nobody → mhoye
I thought the problem had gone away, but duplicated entries just showed up again and not triggered by the addition of any new posts to the blog.

A few days ago I saved a copy of the of the planet feed and the hacks feed to be able to compare them if the problem showed up again (to see if e.g. time stamps had changed). At the time I did not notice that the planet feed did not contain the hacks items. When the duplicated entries showed up just now, I saved a new copy and compared the the old. The hacks-feed had not changed, and neither had the hacks-items in the planet-feed (still not there). A few minutes later I saved a new copy and compared and the planet-feed now contained the hacks-items. A few minutes later they are gone again.

So it seems the hacks-items are not always included in the planet-feed. Are there some odd (broken) caching going on? What is different from all other blogs?
Thanks for your analysis - can you pastebin it somewhere? I'm looking into it as best I can from our side, but Planet's software is known to be brittle in a couple of exciting ways.
I don't think there is much to pastebin. The (currently four) items from the hacks blog are simply left out of the feed with no other changes. As I write this the items are missing again.

The url that I am talking about, by the way, is http://planet.mozilla.org/projects/rss20.xml (it occurred to me that I had forgotten to give it).

You might want to set up an alert using e.g. gotsitemonitor.com (it is free) to check the url for the text "<title>hacks.mozilla.org:" to get an idea of how often the problem happens. I just set one up out of curiosity, but if you have access to the server, you might be able to use it for something.
The monitoring (with a check every 5 minutes) shows quite a different pattern than I expected. The entries from the hacks blog seem to appear and disappear quite often. It seems odd that the file (I assume it is a static file and there is only one server) is regenerated that often:

Start (UTC)           End (UTC)           Minutes State
2015-01-13 10:44:44   2015-01-13 11:19:45    35   Fail
2015-01-13 10:24:39   2015-01-13 10:39:36    15   OK
2015-01-13 10:09:47   2015-01-13 10:19:42    10   Fail
2015-01-13 09:54:39   2015-01-13 10:04:39    10   OK
2015-01-13 09:24:42   2015-01-13 09:49:41    25   Fail
2015-01-13 09:09:40   2015-01-13 09:19:37    10   OK
2015-01-13 08:39:41   2015-01-13 09:04:46    25   Fail
2015-01-13 08:24:43   2015-01-13 08:34:35    10   OK
2015-01-13 08:09:44   2015-01-13 08:19:45    10   Fail
2015-01-13 06:59:37   2015-01-13 08:04:37    65   OK
2015-01-13 06:39:42   2015-01-13 06:54:42    15   Fail
2015-01-13 06:14:36   2015-01-13 06:34:36    20   OK
2015-01-13 05:44:44   2015-01-13 06:09:43    25   Fail
2015-01-13 05:09:38   2015-01-13 05:39:36    30   OK
2015-01-13 04:29:44   2015-01-13 05:04:42    35   Fail
2015-01-13 03:54:43   2015-01-13 04:24:35    30   OK
2015-01-13 03:39:44   2015-01-13 03:49:42    10   Fail
2015-01-13 03:09:43   2015-01-13 03:34:35    25   OK
2015-01-13 02:59:46   2015-01-13 03:04:45     5   Fail
2015-01-13 02:24:44   2015-01-13 02:54:37    30   OK
2015-01-13 00:39:44   2015-01-13 02:19:41   100   Fail
2015-01-13 00:29:38   2015-01-13 00:34:37     5   OK
2015-01-12 23:44:43   2015-01-13 00:24:40    40   Fail
2015-01-12 23:24:46   2015-01-12 23:39:34    15   OK
2015-01-12 22:39:44   2015-01-12 23:19:44    40   Fail
2015-01-12 22:24:39   2015-01-12 22:34:38    10   OK
2015-01-12 21:39:42   2015-01-12 22:19:42    40   Fail
2015-01-12 21:09:43   2015-01-12 21:34:34    25   OK
2015-01-12 20:54:43   2015-01-12 21:04:41    10   Fail
2015-01-12 20:39:37   2015-01-12 20:49:36    10   OK
2015-01-12 20:09:42   2015-01-12 20:34:42    25   Fail
2015-01-12 19:54:44   2015-01-12 20:04:36    10   OK
2015-01-12 19:29:41   2015-01-12 19:49:42    20   Fail
2015-01-12 19:09:41   2015-01-12 19:24:35    15   OK
2015-01-12 18:09:41   2015-01-12 19:04:45    55   Fail
2015-01-12 17:06:02   2015-01-12 18:04:35    59   OK
2015-01-12 16:36:09   2015-01-12 17:00:57    25   Fail
2015-01-12 16:26:00   2015-01-12 16:30:53     5   OK
OK, I can't reproduce this. But more importantly, I can't see why it would occur at all. Feeds from hacks.mozilla.org aren't sent directly to Planet - they end up on Projects. 

I'm strongly tempted to blame this on your Thunderbird setup, to be honest. Can you tell me about that at all? Anything special about it?
Flags: needinfo?(andershol)
Which part can't you reproduce? I thought using an external service removed my setup from suspicion. I am also observing the problem in the browser (you can't see the raw feeds in Thunderbird), so I can't see how this could be a Thunderbird problem.

I tried to set up monitoring of https://hacks.mozilla.org/feed/ checking for "2015" which ought to be found in the lastBuildDate-element. But it only occurs about half the time. When checking about every 5 minutes the state changes ("2015" appears/disappears) every 2.2 checks on average. I've see the build dates of "Wed, 26 Nov 2014 16:42:39 +0000" and "Mon, 12 Jan 2015 19:23:21 +0000" but haven't checked that those are the only ones. So I now suspect that the problem lies in a broken proxy/loadbalancer in front of the hacks-blog. I suspect it is related to https://wiki.mozilla.org/HacksPostMigrationUserDetails based on when the problem seem to have started.
Flags: needinfo?(andershol)
> Feeds from hacks.mozilla.org aren't sent directly to Planet - they end up on Projects.
Yes, see the url in comment 4. I realize I had been imprecise (but not, I think, wrong) up til then.
Also note that in the two versions of the hacks-feed I see, the generator-field is "http://wordpress.org/?v=4.0.1" and "http://wordpress.org/?v=4.1" respectively, so something might still point to an old installation. But the site seems to be both behind both some dns-balancing and load-balancing, so the problem could lie several places.
OK, I've dumped the old Hacks feed, and have re-added the RSS20.xml feed mentioned above. Let's see what happens?
I understand that this is functionally equivalent to saying "I rebooted", but bear with me here.
Thanks for trying. I assume comment 10 talks about the cache of the hacks-feed on the planet-server. After the test mentioned in comment 7 I do however believe the problem lies with the hacks-site, so I think I'll try to open a bug on that.
Depends on: 1122247
(In reply to Mike Hoye [:mhoye] from comment #10)
> OK, I've dumped the old Hacks feed, and have re-added the RSS20.xml feed
> mentioned above. Let's see what happens?

The two latest entries (e.g. "Air Mozilla: ThinkFWD ...") are prefixed with 13 * "hacks.mozilla.org: ". Might this be related to that change or is that anther bug?
Backed that change out because OMGWTF and possibly also BBQ. This isn't a priority right now, but I'm still working on it.
Hey, not sure if this helps at all. BUT maybe it has something to do. When visiting the hacks site itself, I sometimes get differing results on the home page. Sometimes the last article we just published is there, as it should be, on the front page. Sometimes it's not.

I wonder if this is a case of a CDN that takes too long to invalidate, confusing both humans and planets? Maybe the symptoms are related.
Craig: Can you help here?
Note that I don't think that I have observed the problem for the hacks blog recently. See also the linked bug 1122247 that I think points to the cause. But that bug may also be out of date.
(In reply to Anthony Ricaud (:rik) from comment #16)
> Craig: Can you help here?

Hm... nope. I'm baffled. I also can't reproduce the error, the only feed I can get is correct and up to date. There's nothing in the theme that's messing with feeds and I don't think any of the plugins are interfering. So my best shot-in-the-dark guess is that it's some cache weirdness beyond my comprehension. If that cache weirdness is on the feed end rather than the planet end, I'd defer to WebOps (jd cced).
We know that the problem here is not with the hacks blog, it's with the Planet aggregator. Webops is aware that we have a problem, but the existing documentation for the stuff underpinning Planet is excitingly not-good.
Doesn't seem to be a problem anymore.
Status: UNCONFIRMED → RESOLVED
Closed: 9 years ago
Resolution: --- → WORKSFORME
Can you clarify what "works for me" means? I backed out hacks' participation in Planet proper, so errors wouldn't appear there. Did you run the test again?
It means that it has been a while since I have observed the problem and if there is no problem there is no reason keeping the bug open (and I seem to have been the only one that have ever seen the problem). I don't know what you mean by "hacks' participation in Planet proper".

After I looked into it and observed what I described in comment 7 and comment 9 and in bug 1122247, I don't think this have ever been a problem on the planet-side (comment 15 sound to concur). I don't know what you based your statement in comment 19, that this was known to be a problem in the aggregator, on.

In comment 3 and comment 19 it sounds like you think the planet software have problems in general. I tried looking for the software and found a svn-repo. Maybe it would be an idea to throw it in on github, list the bugs you want fixed in a readme and link to the source from the planet page (more prominently than the far bottom).
Reopening because hacks.mozilla.org has not been added back to planet.
Status: RESOLVED → REOPENED
Ever confirmed: true
Resolution: WORKSFORME → ---
When was it removed? Comment 14 could be read that way, but it was over a month after comment 13, so it was probably the fix.
It was removed in r137178 and that is the last commit as of today: http://viewvc.svn.mozilla.org/vc/projects/planet/branches/projects/?view=log
I've re-added the hacks blog to Projects, and if it doesn't lose its mind there I'll re-add it to Planet proper.
... aaaaand, backed it out.
If you compare config.ini revision 135857 (i.e. right before the Jan 15th backout) to revision 139041 (i.e. right before the latest backout), you get:

-[http://hacks.mozilla.org/feed/]
+#pulling this until we understand what's going on.
+[http://planet.mozilla.org/projects/rss20.xml]
 name = hacks.mozilla.org

That is you have been adding the planet feed back (which ought to be weird), not the hacks feed.
Huh, right. Let me try again....
(In reply to Anthony Ricaud (:rik) from comment #25)
> It was removed in r137178 and that is the last commit as of today:
> http://viewvc.svn.mozilla.org/vc/projects/planet/branches/projects/?view=log

Thanks for providing the link.

(In reply to Mike Hoye [:mhoye] from comment #29)
> Huh, right. Let me try again....

A  number of about 10 hacks posts where added, so it seems to have worked. You probably should remove the comment from the config since it is no longer true.
Can we mark this as resolved now that bug 1122247 have been fixed and the feed have been re-added?
Flags: needinfo?(mhoye)
Hasn't recurred in a long time, so closing.
Flags: needinfo?(mhoye)
To be clear: I am not happy with that resolution, but I don't see any other options, so.
Status: REOPENED → RESOLVED
Closed: 9 years ago → 8 years ago
Resolution: --- → FIXED
You need to log in before you can comment on or make changes to this bug.