523865 - Stats pages are running out of memory again.

Status: VERIFIED FIXED

(addons.mozilla.org Graveyard :: Statistics, defect)

Description

[Wil Clouser [:clouserw]]

Case in point: https://addons.mozilla.org/en-US/statistics/csv/1865/application?group_by=date

This is going to continue to happen, so last month we had a discussion to get ideas on fixes: https://wiki.mozilla.org/Webdev:Meetings:2009-09-01

I'll ask for infra's input and then we'll make a decision.

Comment 1

[Justin Fitzhugh]

rather than add memory for this one specific use case to all the webheads, seems more logical to limit the size of the date range or plot points.  someone who really wants all the data can make a few requests over multiple ranges.

Comment 2

[Wil Clouser [:clouserw]]

(In reply to comment #1)
> rather than add memory for this one specific use case to all the webheads,
> seems more logical to limit the size of the date range or plot points.  someone
> who really wants all the data can make a few requests over multiple ranges.

No worries, adding RAM wasn't on our list.  I'll write a patch that will give a maximum of 6 months of data when requested.

Comment 3

[Wil Clouser [:clouserw]]

Attachment: add a six month cap

This adds a start/end date to all the stats queries save a couple contribution ones that I think are only on the admin panel.

This has the side effect of everwhere that we say "All Time" in the stats panel it's actually 6 months.  "All Time" is still a useful number though, so I think we should be calculating that offline so we can still provide it (more cron jobs!).  If people agree with that, please file another bug.

Comment 4

[Wil Clouser [:clouserw]]

CCing fligtar & osunick so they are aware of the number changes in comment 3

Comment 5

[Fred Wenzel [:wenzel]]

Comment on attachment 408860 [details] [diff] [review]
add a six month cap

Looks good. r+ed under the assumption that your reduction to 6 months tops is acceptable.

Comment 6

[Justin Scott [:fligtar]]

Hmm... it's pretty important to have all the data available for the graphs. I don't care so much about the CSV files, except that the graphs pull data from the CSV files.

Comment 7

[Wil Clouser [:clouserw]]

Thanks wenzel, r54477

If you want complete info on the graphs we should be taking weekly averages or something.  Plotting daily points for several years worth of data is taxing on the server and the client side.

QA:  URLs like https://preview.addons.mozilla.org/en-US/statistics/csv/1865/application should return CSVs.  Passing start and end through the URL should adjust what you see.  You shouldn't be able to retrieve more than 6 months of data.

Comment 8

[Stephen Donner [:stephend] Not actively reading bugmail]

Verified FIXED; I tested:

* https://preview.addons.mozilla.org/en-US/statistics/csv/1865/application
* https://preview.addons.mozilla.org/en-US/firefox/statistics/csv/1865/application?start=2009-04-30&end=2009-10-30
* https://preview.addons.mozilla.org/en-US/firefox/statistics/csv/1865/application?start=2001-04-30&end=2009-10-30 ("Error: The requested data is too broad. Dates can be a maximum of six months apart.")
* https://preview.addons.mozilla.org/en-US/firefox/statistics/csv/1865/application?start=2009-04-30&end=2009-11-30 ("Error: The requested data is too broad. Dates can be a maximum of six months apart.")
* https://preview.addons.mozilla.org/en-US/firefox/statistics/csv/1865/application?start=2009-04-30&end=2009 ("invalid date format")