704379 - Problems fetching correlation reports should be logged as warning not error

Status: RESOLVED FIXED

(Socorro :: General, task)

Description

[Robert Helmer [:rhelmer]]

Now that we push PHP app logs to Arecibo (bug 675985), we get ~300 errors per topcrash page, when the product/version on that page does not have valid correlation reports.

We're working on integrating correlation reports directly into Socorro, but in the meantime let's just turn down the logging to "warning" (we only send "error" to Arecibo).

Comment 1

[Robert Helmer [:rhelmer]]

lonnen, brandon - r? https://github.com/mozilla/socorro/pull/171

I am not entirely pleased with this approach. The problem is that gzopen/fopen don't throw exceptions, but they honor set_error_handler.

Based on comments in the ErrorHandler class it was once used for getJsonZ but this no longer appears to be the case (presumably this was moved to the middleware and uses something other than fopen/gzopen now).

This patch checks the error string for "HTTP/1.1 404 NOT FOUND" and logs warning in that case, error otherwise.

The only alternative I can think of (besides switching the whole thing to another lib like curl) is to do an @get_headers() right before the fopen/gzopen, and abort if there is a 404. The requires an extra HTTP request per correlation report, which normally would not bother me much but as I mentioned we do 300 of these per topcrash report :(

Comment 2

[Lonnen :lonnen]

It's not pleasant, but its Kohana-ish (given how logging works, yeah?). Warnings seems sufficient for this, since this problem is already well handled.

Comment 3

[[github robot]]

Commits pushed to https://github.com/mozilla/socorro

https://github.com/mozilla/socorro/commit/6f1d60fcc330eedb1ce34be49b733a477cc116d9
bug 704379 - log as warning not error for correlation report if HTTP fetch failure is due to 404

https://github.com/mozilla/socorro/commit/d86967287651af34e5d982907f22ccceb863260d
Merge pull request #171 from rhelmer/bug704379-correlation-warning

bug 704379 - log as warning not error for correlation report if HTTP fet...

Comment 4

[Robert Helmer [:rhelmer]]

Thanks, seems to work on crash-stats-dev!

One interesting thing - now that these are out of the way, I see we're sometimes getting some other interesting entries on crash-stats-dev:

"""
fopen(http://people.mozilla.com/crash_analysis/20111121/20111121_Firefox_3.5.13-interesting-addons.txt): failed to open stream: operation failed /data/socorro/htdocs/application/libraries/Correlation.php line 112

gzopen(): Failed to enable crypto /data/socorro/htdocs/application/libraries/Correlation.php line 112

gzopen(): SSL: connection timeout /data/socorro/htdocs/application/libraries/Correlation.php line 112
"""

I think we're just making *way* too many requests... in fact something very funky is going on in this code, because I just counted *1800* 404 warnings logged for a single topcrasher page load (!)

I've filed bug 704410 to track that.