493965 - Create feed for press news

Status: VERIFIED FIXED

(www.mozilla.org :: General, defect)

Description

[Samuel Sidler (old account; do not CC)]

We should have a feed for the press news page.

See also bug 439539 comment 3.

Comment 1

[Stephen Donner [:stephend] Not actively reading bugmail]

We really should fix this or remove the link (let's not do the latter).

Comment 2

[John Slater]

Shannon Prior and Les Orchard are already working on this project, actually. But, it's a Q3 project, not a 3.5 one, so I'm removing that milestone.

Comment 3

[Les Orchard [:lorchard]]

Attachment: Quick & dirty feed scraper for press releases

Here's a stab at generating an RSS feed for the en-US press releases page, along with header links on /en-US/press/ and /en-US/press/news.html

It's a little crude, but works and doesn't require a DB.

Comment 4

[Wil Clouser [:clouserw]]

That's an interesting idea which should work.  Can you change the patch to write to a file instead of just outputting so we can do it on a cron?  Even if it's generated every couple of minutes I'd be happier sending a static file then reading the dir tree every time.

Comment 5

[Wil Clouser [:clouserw]]

Comment on attachment 390489 [details] [diff] [review]
Quick & dirty feed scraper for press releases

r+ for the idea, but just need it writing to a file instead of live.  Thanks.

Comment 6

[Les Orchard [:lorchard]]

Hmm, I'd thought about a cron job, but I'm not familiar at all with the mozilla.com hosting setup.  Should the files just get written to the press directory?  Is there a single origin server / NFS mount for all of mozilla.com, or is there a cluster?  Was concerned about complexity there.

Don't we have a frontend cache on mozilla.com?  Seems like that would keep the scraper from actually getting hit much.  The script itself could cache in memcache, too.  Beyond that, reading a directory and a few files isn't expensive when it does get hit.

Comment 7

[Wil Clouser [:clouserw]]

I'll CC oremj for any infra questions.

There is a front end cache and it would probably work fine.  It's just that moz.com gets so much more traffic than other sites I'm overly paranoid about it.  If IT is happy with the patch then it's fine with me too.

Comment 8

[Jeremy Orem [:oremj]]

It's a cluster. I'd run the cron job on the admin server, which would then sync out to the cluster. If the files are being read locally the frontend cache should be fine as far as load goes.

Comment 9

[Les Orchard [:lorchard]]

Which would be easier to maintain?  

I could wrap the PHP in output buffering and write to file to make it work in a crontab.

Or, I can leave it as-is and rely on the frontend cache.  It reads around 7 files from its local directory and sets Expires and Cache-Control headers for an hour, which I assume Zeus (?) honors.

Comment 10

[Les Orchard [:lorchard]]

Ping?  Does this still need revising as a cron-job, or fine as-is?

Comment 11

[Les Orchard [:lorchard]]

Comment on attachment 390489 [details] [diff] [review]
Quick & dirty feed scraper for press releases

Is this good as-is, or does it still need reworking as a crontab? oremj seems okay with it

Comment 12

[Wil Clouser [:clouserw]]

Comment on attachment 390489 [details] [diff] [review]
Quick & dirty feed scraper for press releases

Like I said in comment 7, if IT is happy with it, I'm happy with it.

Comment 13

[Michael Morgan [:morgamic]]

Jeremy - review it!

Comment 14

[Les Orchard [:lorchard]]

Before this patch gets any older, I checked it in as r49639.  Let me know if it needs any tweaks

Comment 15

[Reed Loden [:reed]]

(In reply to comment #14)
> Before this patch gets any older, I checked it in as r49639.  Let me know if it
> needs any tweaks

You landed it directly on stage... It needs to be on trunk.

Comment 16

[Les Orchard [:lorchard]]

Hmm, not exactly sure how I managed that, but: r52116

Comment 17

[Stephen Donner [:stephend] Not actively reading bugmail]

Verified FIXED: https://www-trunk.stage.mozilla.com/en-US/press/feed.php