Closed Bug 692035 Opened 13 years ago Closed 13 years ago

mozilla.com BBC feed stopped working and is now 404

Categories

(mozilla.org Graveyard :: Server Operations, task)

Product:
mozilla.org Graveyard
Component:
Server Operations
Platform:
x86_64
Windows 7
Type:
task
Priority:
Not set
Severity:
normal

Tracking

(Not tracked)

Status:
RESOLVED FIXED

People

(Reporter: nigelh747, Assigned: nmaul)

Details

User Agent: Mozilla/5.0 (Windows NT 6.1; WOW64; rv:7.0.1) Gecko/20100101 Firefox/7.0.1
Build ID: 20110928134238

Steps to reproduce:

Started Firefox - in normal and safe modes with different users and looked at the headlines - that is set to that of the BBC


Actual results:

Message - Live Bookmark failed to load - last update 3 October 2011


Expected results:

Bookmark should have updated with proper feed

Feed Location
http://en-gb.fxfeeds.mozilla.com/en-GB/firefox/headlines.xml

Site Feed
http://www.bbc.co.uk/go/rss/int/news/-/news/
Undertook a manual Reload Live Bookmark - and it continued to fail.
Component: General → Bookmarks & History
If I open the feed location I get a 404 error
It's possible this is an issue with the mozilla.com => mozilla.org rewrite
Component: Bookmarks & History → Webdev
Product: Firefox → mozilla.org
QA Contact: general → webdev
Version: 7 Branch → other
Summary: Live Bookmark Feed Failed to Load BBC → mozilla.com BBC feed stopped working and is now 404
What does the fxfeeds subdomain do? I have no idea how that relates to the merge. We only switched mozilla.com, not any subdomains.
Shooting this over to IT: Can you investigate what this server is supposed to come from and why it doesn't? As an in-product URL, I also imagine we might have (or need) nagios checks for this.

As far as I can tell, this service is not part of the mozilla.com codebase and thus should be unaffected by the merge.
Assignee: nobody → server-ops
Component: Webdev → Server Operations
QA Contact: webdev → cshields
(In reply to Fred Wenzel [:wenzel] from comment #5)
> Shooting this over to IT: Can you investigate what this server is supposed
> to come from

"what server this is supposed to come from"
fxfeeds.mozilla.com/org was not part of the www.mozilla.com/org merge.

However, in the last few days we have done quite a bit of work on fxfeeds.mozilla.org, in order to reduce the load we receive from this site (quite substantial). Earlier this week (Monday or Tuesday, I think), we moved it from an entirely local site to being hosted on Edgecast CDN.

The two links in comment 0 are both working for me, and my 'Latest Headlines' bookmark seems to be working properly as well.

Perhaps this was a transient issue at the BBC?
Just tried again and the situation continues...
With the same two links in comment 0? Which link is throwing the 404?

It's worth noting that for me the first link redirects to this:
http://feeds.bbci.co.uk/news/rss.xml?edition=uk

Not your second link. Perhaps BBC changed their link format and your browser is caching an old redirect? Might try flushing your cache and seeing if that helps.

Either way, we might want to see about changes to the way this feature works, such that if it gets a 404 it has some sort of sane fallback mechanism. We are ultimately at the mercy of 3rd-party services here, and if they make changes unexpectedly it can and will break this feature.
We have discovered the issue. It was caused by a change we made earlier this week, affecting fxfeeds.mozilla.com. We moved this onto a CDN to improve performance and reduce load on our servers.

In my Firefox, the Feed Location is this:
http://fxfeeds.mozilla.com/en-US/firefox/headlines.xml

This URL works properly. You can change en-US to any other locale and it continues to work as expected (does not 404).

However, it seems that in your copy, the domain name is actually en-gb.fxfeeds.mozilla.com. I don't know if this is different in different builds, or if it's an artifact from old versions being upgraded and the URL not getting updated.

This extra subdomain level is what's breaking it. The CDN does not support wildcard subdomains. We could enumerate all possible locales that are relevant, but that's likely to be a maintainability nightmare, so instead we are changing this so that links like this don't go to the CDN at all, but come straight to us (effectively restoring the old behavior).

The fix is in place, but due to caching (both browser cache and DNS record propagation), you may not be able to see it right away. I suggest you open and close Firefox, and/or reboot. Either of these should be enough to resolve any problems on your local machine, but unfortunately won't help if your ISP is still caching the bad records. You can simply wait for them to time out (which will be any time in the next 24 hours), or you could change the Feed Location. Simply remove 'en-gb.' from the start of the domain, and it should start working again.

I'm going to keep this bug open for a day or so, while the new DNS records propagate out.


I am sorry for the inconvenience, and thank you for the bug report... the info in  your original description was instrumental in locating the problem.
Assignee: server-ops → nmaul
Status: UNCONFIRMED → ASSIGNED
Ever confirmed: true
 (In reply to Jake Maul [:jakem] from comment #10)
> We have discovered the issue. It was caused by a change we made earlier this
> week, affecting fxfeeds.mozilla.com. We moved this onto a CDN to improve
> performance and reduce load on our servers.
> 
> In my Firefox, the Feed Location is this:
> http://fxfeeds.mozilla.com/en-US/firefox/headlines.xml
> 
> This URL works properly. You can change en-US to any other locale and it
> continues to work as expected (does not 404).
> 
> However, it seems that in your copy, the domain name is actually
> en-gb.fxfeeds.mozilla.com. I don't know if this is different in different
> builds, or if it's an artifact from old versions being upgraded and the URL
> not getting updated.
> 
> This extra subdomain level is what's breaking it. The CDN does not support
> wildcard subdomains. We could enumerate all possible locales that are
> relevant, but that's likely to be a maintainability nightmare, so instead we
> are changing this so that links like this don't go to the CDN at all, but
> come straight to us (effectively restoring the old behavior).
> 
> The fix is in place, but due to caching (both browser cache and DNS record
> propagation), you may not be able to see it right away. I suggest you open
> and close Firefox, and/or reboot. Either of these should be enough to
> resolve any problems on your local machine, but unfortunately won't help if
> your ISP is still caching the bad records. You can simply wait for them to
> time out (which will be any time in the next 24 hours), or you could change
> the Feed Location. Simply remove 'en-gb.' from the start of the domain, and
> it should start working again.
> 
> I'm going to keep this bug open for a day or so, while the new DNS records
> propagate out.
> 
> 
> I am sorry for the inconvenience, and thank you for the bug report... the
> info in  your original description was instrumental in locating the problem.

Not a problem - to confirm this now works - do you want to close this?
Yep, it should be pretty well propagated now. Closing.

To clarify the ultimate solution, we did eventually resort to enumerating the top few locales, and sending them to the CDN. The rest come directly to us. This is apparently more common than we had previously believed. My belief is it's something that was commonly done in earlier versions, and is not changed on updates.
Status: ASSIGNED → RESOLVED
Closed: 13 years ago
Resolution: --- → FIXED
The issue is still occurring with this feed URL:

http://fxfeeds.mozillazine.org/rss20.xml    (HTTP 404)

The Google Reader history for the feed confirms it was working until 5 days ago.
(In reply to davidtryse+bz from comment #13)
> The issue is still occurring with this feed URL:
> 
> http://fxfeeds.mozillazine.org/rss20.xml    (HTTP 404)
> 
> The Google Reader history for the feed confirms it was working until 5 days
> ago.

Mozillazine is not run by the Mozilla Foundation (confusing, I know), so if you'd like to report a bug to them, you'll need to contact them directly.

http://www.mozillazine.org/about/
Hi Fred,

Look like they bounced it back to Mozilla actually :)
per the latest comment here:
http://forums.mozillazine.org/viewtopic.php?p=11346557#p11346557
It's not 404'ing anymore for me. If they're just pulling from us somehow, they may have been doing so in a way that was affected by the issue mentioned above. However I don't have access to their code / config, so I couldn't say for sure how exactly they're getting that feed (what URL they hit).

In any case, seems resolved now. If it's not, we'll need the Mozillazine folks to let us know what URL of ours is not working. Thanks!
Hi Jake,

Odd..I still get 404 over here for http://fxfeeds.mozillazine.org/rss20.xml
(Google Reader also still says fetching error)

http://redbot.org/?uri=http://fxfeeds.mozillazine.org/rss20.xml

I'll check Mozillazine as suggested.
Product: mozilla.org → mozilla.org Graveyard
You need to log in before you can comment on or make changes to this bug.