Clean out old and un-used feeds to prevent errors


Code Cleanup
3 years ago
3 years ago


(Reporter: groovecoder, Unassigned)



The only feed items we display on the site are from Hacks. We can remove all the other bundles and feeds. Especially now that many are causing errors ...

Old and un-used:
Unable to fetch

Removed/missing feeds (failing since the move of to WPEngine):
Unable to fetch
Unable to fetch
Unable to fetch
Hmm ... All of these feeds work except for the MDN one, which I deleted from the stage and prod servers.

:cyliang - can you see any reason why the stage & prod cron hosts wouldn't be able to reach those and urls? Maybe a netflow needs to be open or something? I'm still fine removing them, would just like to know why they're failing.
Flags: needinfo?(cliang)

Comment 2

3 years ago
I’m not sure why the feed links are failing.  A quick test shows that they should, for the most part, work when invoked from the developer web heads. [1]

Some questions:

  * How are the feed links being parsed?  Is there some regular job that grabs new feeds?  (I’m trying to see if I should be testing feed access from the celery nodes or some other part of the MDN infra.)
  * Do you know what user agent string you are emitting?  (I can try running a wget that spoofs that agent.)  In, we discovered that wp-engine apparently filters out UA strings it considers “harmful”.  >_< updated their code to emit a different user agent string.

[1] When I attempt to grab the feeds via wget from the developer webheads, I succeed for the http feeds but not the https ones.  That, I think, has to do with a bug in the version of wget running on those servers.  (I can successfully grab the HTTPS feed from developeradm, which is running a newer version of wget.)
Flags: needinfo?(cliang)
* There's an 'update_feeds' cron job, which I think runs on the admin node?

* Our feedparser package uses urllib2, which uses "Python-urllib/2.6" by default [1] as the user agent.

I updated all the stage feeds [2] to https if possible. does not respond on https. I deleted all the twitter feeds, as they're not available anymore.

I'll keep an eye on the update_feeds emails to see if these changes fix the errors.

The error was down to only the http so I removed that from stage too.
Since I removed the planet feed, we're not getting so many errors. Still getting this one intermittently:

http_app_kuma: 2015-05-14 08:30:15,858 kuma.feeder:ERROR Unable to fetch Exception: <urlopen error The read operation timed out>: /data/developer/www/
You need to log in before you can comment on or make changes to this bug.