519871 - Get collector traffic (wc -l) over last 3 months

Status: RESOLVED FIXED

(Mozilla Metrics :: Data/Backend Reports, defect)

Description

[Michael Morgan [:morgamic]]

We're trying to profile crash volume and need to run a line count on access logs dating back as far as we can go.  You guys have a script for this -- can we run this on crash-reports.m.... and get a simple line graph of count per day?

Comment 1

[matthew zeier [:mrz]]

Can Metrics help here?

Comment 2

[Pedro Alves]

that should be fairly simple, where do the logs sit and what in the logs shuold we be looking for? (or are we looking for everything in the wc -l style?)

Comment 3

[Michael Morgan [:morgamic]]

Ideally we'd find a way to trend these logs over time, and if you have the ability to split by product, version and platform that's a definite bonus.

Comment 4

[Pedro Alves]

Tell me where that is and I'll see what I can do.


Full analysis on crash data is a big Q4 objective for us, but if I can quickly come up with something to to solve this particular questions, happy to help

Comment 5

[Daniel Einspanjer [:dre] [:deinspanjer]]

In about half an hour, the following file will be finished writing out:
cm-metricsetl02:/etl/processed/crash_report_trend.tsv.gz

It contains three columns,
yyyy-mm-dd-hh    user_agent_string    count_requests


Currently, we're averaging about 150 unique user agent strings.  The Mac crash reporters tend to put the precise name of the machine type (i.e. iMac MacBook MacBookPro) plus the kernel version number which leads to fragmented records.

It is likely the file might be useful to you as is, but if not, Pedro or I can easily take it and run a real ETL process on it to properly tokenize and cluster the UA strings.  Just let us know what type of analysis you want to do.

For reference, here is the one-liner I used to create the file.  It probably would have been a little easier to read if I had done a true perl script but...

zgrep -oP '(?<=" ")[^"]+(?=" ")' *.gz | pv -s $(gzip -l *.gz | tail -1 | awk '{print $2}') | sed -re 's/^access_(.............)[^:]+:(.*)/\1\t\2/' | sort -S 10G | uniq -c | awk '{printf "%s\t%s\t%d\n", $2, $3, $1}' | gzip > /etl/processed/crash_report_trend.tsv.gz

zgrep -oP
      -- only print the matching text, use Perl like syntax
'(?<=" ")[^"]+(?=" ")'
      -- A regex to match just the user agent string without the surrounding quotes (so -o gives me what I want)

pv -s $(gzip -l *.gz | tail -1 | awk '{print $2}')
      -- pipeviewer to give me status and performance of the oneliner.  Since I can't muck with the list of filenames that zgrep is parsing (otherwise, it wouldn't emit the "filename:" prefix), I manually calculate the total size of the stream with gzip -l

sed -re 's/^access_(.............)[^:]+:(.*)/\1\t\2/'
      -- Strip off the "filename:" part returning just the yyyy-mm-dd-hh and the UA

sort -S 10G | uniq -c
      -- Big fat sort buffer, then emit the unique rows with a count

awk '{printf "%s\t%s\t%d\n", $2, $3, $1}'
      -- Reorder the columns and make them tab separated

gzip > /etl/processed/crash_report_trend.tsv.gz
      -- Write it out to a gzip file.

Comment 6

[Daniel Einspanjer [:dre] [:deinspanjer]]

Sorry, one more thing:  The requests don't have much information about the product or version.  If it can't be derived from the information in the user agent string, it won't be able to be derived from the access log requests at all.

Comment 7

[Daniel Einspanjer [:dre] [:deinspanjer]]

Attachment: Number of crash reports by user agent string by yyyy-mm-dd-hh

morgamic:

Could you review the user agent strings captured in this and suggest what kind of breakdowns would be useful to you (or anyone else) on this?

Metrics can work on capturing this data out of the log files on a continual basis, but we need to know the best way to parse this string to allow people to make sense of it.

Comment 8

[Daniel Einspanjer [:dre] [:deinspanjer]]

p.s. the data starts at 2009-09-12-21 because those were the earliest logs I had available.  It looks like the logs get deleted less than a month after creation.

Comment 9

[Michael Morgan [:morgamic]]

Shaver - here's what you asked for.  Daniel - I think Shaver should work with the Firefox team to provide smarter URLs similar to AUS2 since we don't have much useful data in the URL currently (since everything comes via POST).  Mike, is that feasible for 3.6 so we can get more data from logs to cover the 100% of crashes information?

Comment 10

[Michael Morgan [:morgamic]]

Comment on attachment 404855 [details]
Number of crash reports by user agent string by yyyy-mm-dd-hh

This is very useful.  Thanks, Daniel.  In the future we should continue to trend this and if we get more accurate information from Firefox's crash URL we can modify but UA string seems to do the trick.

Comment 11

[Daniel Einspanjer [:dre] [:deinspanjer]]

For now, if someone could review the UA strings that we currently receive and give us an idea of some logic we could apply to categorize them, we can easily turn this one-time processing into a continuously run process that we can trend.

Comment 12

[Michael Morgan [:morgamic]]

Anything with CFNetwork is Mac -- the crash reporter on Mac doesn't properly set the UA string so the default network handler is using its UA.  Unfortunately there's a lot of variance here, but we don't have product info.

Linux and Windows are easily identifiable:
   283 2009-09-12-22   Breakpad/1.0 (Linux)    39
   284 2009-09-12-22   Breakpad/1.0 (Windows)  27611

So, I think the best we can do now is crash volume over time per platform -- and you could get that by grouping CFNetwork stuff under the Mac umbrells and using Breakpad/1.0 ($platform) for the other two.

Comment 13

[Mike Shaver (:shaver -- probably not reading bugmail closely)]

Adding Ted, re comment 9.

Comment 14

[(not currently active) Ted Mielczarek]

Filed bug 521349 for the client enhancement. Please add any requirements there.

Comment 15

[Daniel Einspanjer [:dre] [:deinspanjer]]

We delivered what we could for the time being.  Reopen if something is missing or file a new bug for future enhancement.