577767 - Data Analysis: Create a feed of problematic URLs

Status: RESOLVED FIXED

(Input :: General, enhancement, P1)

Description

[Fred Wenzel [:wenzel]]

From bug 577294:

Beltzner wrote:

"I think we need to create a table of URLs reported as problematic with:

URL
# of unique reports
first reported date
last reported date
comments (blob)
"

Can probably make this an RSS feed, and either cluster these reports by URL or by domain, sorting it descendingly by report frequency. Could create this report once a day (or something like that) and cache it.

Comment 1

[Aakash Desai [:aakashd]]

Attachment: mock up

The buttons to get to the page aren't put onto the mock up, but essentially it'd be a part of the header somewhere...and, no, we're not removing the Input Dashboard header image.

Comment 2

[Aakash Desai [:aakashd]]

Attachment: better mock up

Comment 3

[Aakash Desai [:aakashd]]

Attachment: site problems mock up

Comment 4

[Aakash Desai [:aakashd]]

Looks like this is missing 1.6. I'm moving this to 1.6.1

Comment 5

[Fred Wenzel [:wenzel]]

mkurze's patch looks good, and I think we can land it soon on stage. However, I don't know where to find some test data. Is there any on cm-webdev01-master01?

Since you set allow_syncdb to False, how do we initially create the schema?

Also, if that hasn't happened yet, please file an IT bug to have them set up the settings necessary to access a copy (presumably) of some site data. They might want to set up a separate database and just import a static dump. If you need them to run a cron job instead, we need to let them know that as well.

Comment 6

[Michael Kurze [:michaelk]]

that's a line that's missing in README.md I am afraid
./manage.py syncdb --database=website_issues

Also, we got access to a daily dump on cm-metricsetl02 and a separate database on cm-metrics-mysql01.

Comment 7

[Michael Kurze [:michaelk]]

To get initial data in, you can run the job on your (dev) machine afterwards.

./manage.py generate_sites

(In the actual setup this will be run on cm-metricsetl02 and the data will be pushed into the websites_issues database).

You can also configure both django connection aliases ("default" and "website_issues") to point to the same actual DB for simplicity.

I can add crontasks on cm-metricsetl02, so I won't need to ask for that.

Comment 8

[Aakash Desai [:aakashd]]

This is on staging now:

http://input.stage.mozilla.com/en-US/sites