897273 - Count of all 404s now happening on crash-stats prod

Status: RESOLVED FIXED

(Infrastructure & Operations Graveyard :: WebOps: Socorro, task)

Description

[Peter Bengtsson [:peterbe]]

As part of https://bugzilla.mozilla.org/show_bug.cgi?id=787240 we want to put up redirects (permanent) for old URLs that have been left lying around (e.g. old bookmarks). 

Can you produce a tally of all URL hits that have 404'ed on crash-stats.mozilla.com?

If you have a tool for grouping them that would be great but I'd be perfectly happy to write a python script myself. For example, if you can upload a couple of zips of access logs grepped on 404 I can trawl through it with a script.

Comment 1

[Brandon Burton [:solarce]]

(In reply to Peter Bengtsson [:peterbe] from comment #0)
> As part of https://bugzilla.mozilla.org/show_bug.cgi?id=787240 we want to
> put up redirects (permanent) for old URLs that have been left lying around
> (e.g. old bookmarks). 
> 
> Can you produce a tally of all URL hits that have 404'ed on
> crash-stats.mozilla.com?
> 
> If you have a tool for grouping them that would be great but I'd be
> perfectly happy to write a python script myself. For example, if you can
> upload a couple of zips of access logs grepped on 404 I can trawl through it
> with a script.

You can SSH to socorro[1-2].webapp.phx1.mozilla.com via the Global VPN and have permissions that should let you read the logs in /var/log/httpd/crash-stats.mozilla.org/ on both servers.

Comment 2

[Peter Bengtsson [:peterbe]]

Attachment: biggest-404s-junejuly-2013.txt

Comment 3

[Peter Bengtsson [:peterbe]]

Attached all the URLs that 404'ed more than 10 times in June and July 2013. 

Now it's up to me in 787240 to do something about these.