Closed Bug 490918 Opened 16 years ago Closed 16 years ago

Keep track of collection downloads

Categories

(Mozilla Metrics :: Data/Backend Reports, defect)

Product:
Mozilla Metrics
Component:
Data/Backend Reports
Type:
defect
Priority:
Not set
Severity:
blocker

Tracking

(Not tracked)

Status:
RESOLVED FIXED
Milestone:
Unreviewed

People

(Reporter: fligtar, Assigned: fligtar)

References

Details

(Whiteboard: amo; etl)

Currently the only collection AMO has is Fashion Your Firefox, but in mid-May we'll be launching a new feature that lets anyone create collections. These collections will be handled differently, as we won't have the shopping cart functionality that FYF has. We're going to add a parameter to download URLs that come from collections that indicate the add-on download came from a collection. When this happens, the stats_addons_collections_counts table should be updated. We're finalizing the URL format in bug 486768, but it will have a collection_id parameter, and we should keep track of the number of downloads of each add-on from each collection per day. We don't need to do anything with the stats_collections_counts table for this tracking. I will give the exact URL format that will be appearing soon, and we'd really like to have this in place for May 21 when we plan to go live with the new features, to avoid having to back-process even more logs. Thanks!
Okay, we've decided on tacking ?collection_id={234324-2234234-234234-234234} onto the end of the download links. Downloads that come from the Firefox extension/sharing API will have an additional &ref=sharingapi parameter. It would be useful to know the number of downloads that came from web collections versus the API/extension, but that's more of a secondary request that shouldn't block the completion of this bug -- I can file another bug for that if you'd like.
The &ref=sharingapi parameter will actually be changed to &src=sharingapi.
Hi guys, Any update on setting up stats for add-on collections?
Didn't get a response from last comment. Collections launches a week from today.
Severity: normal → critical
Sorry for missing that last comment. I am ready to push the new collection stats processing changes out, but I could use some data to test with. I checked logs from addons.mozilla.org and from preview.addons.mozilla.org and found no requests with the parameter collection_id=. Regarding comment #1, could we get a second bug talking about this sharingapi thing? What exactly it tracks, and what places we want to track it? (i.e. is it for collections only or for any addon that is somehow shared and downloaded?)
I just "downloaded" a few add-ons from 2 different collections before making this comment, so hopefully you'll see some examples in the preview.addons.mozilla.org logs now. As for the second part, I will file a bug later on for that -- it's not important for the short term.
(In reply to comment #1) > Okay, we've decided on tacking ?collection_id={234324-2234234-234234-234234} > onto the end of the download links. > > Downloads that come from the Firefox extension/sharing API will have an > additional &ref=sharingapi parameter. It would be useful to know the number of > downloads that came from web collections versus the API/extension, but that's > more of a secondary request that shouldn't block the completion of this bug -- > I can file another bug for that if you'd like. I'm assuming you're not just using regex but to be clear, those aren't the only two parameters that might be in the URL.
Code is written and tested with the preview.addons.mozilla.org test requests mentioned in comment #6. Deploy will happen this weekend 2009-06-06. Will be marked fixed then. When data goes live, need validation and reopening if any problems.
Assignee: nobody → deinspanjer
Status: NEW → ASSIGNED
Great! Thanks!
fligtar, I've been monitoring the collections traffic on production today, and we have a problem with the stats tracking that I don't know the solution for. I need input from you guys. In the traffic I tested against from preview.addons.mozilla.org, the collections requests followed a consistent pattern: The first request was for /downloads/latest/<addonid>/addon-<addonid>-latest.xpi?collection_id=.* There was an immediate subsequent request for the actual file /downloads/file/<fileid>/xpi/<filename>?collection_id=.* In production, I'm seeing many requests that are missing the second request with the fileid, and a portion of requests that do not have the first request for the /latest/ URL. Right now, the standard addon download tracking ignores the /latest/ requests and only tracks the requests for specific files. Is it acceptable for the collection tracking to do the same given the requests I'm seeing that seem to be not redirecting to the file url?
Assignee: deinspanjer → fligtar
Severity: critical → blocker
Fred, does this have something to do with when the file is served from the webheads in the period before it is pushed to mirrors?
Considering the /latest/ link is just a shortcut that redirects to downloads/file, yes you can probably ignore those altogether. All requests will "come by" downloads/file (so, count this), and then it's either served off the webheads or forwarded to the release mirrors.
So what do you think might be happening with the large number of requests I'm seeing that have /downloads/latest/ but no corresponding /downloads/file/ ?
I can only guestimate, judging by the code, that these don't have a second request because either the file does not exist or they do not actually download the file they are redirected to. Are these bots? Also, did you ever try one of these suspicious links? Do they return actual xpi files or an error message?
When I try the links on preview, and on AMO, they redirect to the file URL. The user agent string doesn't look anything like a bot and other requests to /latest/ from the same IP will have successful redirects. For now, We'll just continue as is, but we need to think about a way to keep track of the problem in case it gets worse.
I seem to remember darkly that the client (Firefox, most likely) queries the same URI multiple times just to download an add-on once. Justin may remember that better; is this still present or is that long gone?
I think it still does. Perhaps the first request is when the xpinstall dialog is popped open, and if the user clicks cancel, it doesn't send additional requests.
We don't see to have any data from yesterday of add-on downloads from collections. Is the script working properly?
The problem mentioned above with the download/latest vs download/file caused the numbers to be skewed. I worked on correcting that last night and will get the updated data into your table today.
fligtar: The data is sitting in the metrics table ready to be exported, but we have one more small problem. Originally, the stats_collections_counts and stats_addons_collections_counts tables were designed to contain the information for the FYF collection mechanism. Back then, I was told that the collection_id would be an integer and that the fyf collection_id was hardcoded to 1. So those two tables on tm-amo01-master01 are ints. With the new collections, collection_id is a GUID. Do I need to pull some mapping table to map the GUID to an int or can you guys change the stats tables to make collection_id a varchar or char instead?
Whiteboard: amo; etl
Scratch comment #20. I logged into the DB and found the collections table has an integer ID for each uuid. So I changed the ETL to join against that table and the data is now exported.
Please check out the data for 2009-06-10 and close if acceptable.
Cool, everything looks good in our db. So this will automatically update every night when the other scripts are run? And will it pick up yesterday's tonight? Thanks!
Status: ASSIGNED → RESOLVED
Closed: 16 years ago
Resolution: --- → FIXED
We have to catch up first. It will update yesterday's numbers in a couple of hours. Tonight's numbers will happen tomorrow afternoon, then after that point it will be automated.
You need to log in before you can comment on or make changes to this bug.