bugzilla.mozilla.org has resumed normal operation. Attachments prior to 2014 will be unavailable for a few days. This is tracked in Bug 1475801.
Please report any other irregularities here.

Support light weight themes with our new metrics solution (HIVE)

RESOLVED FIXED in 2014-09

Status

addons.mozilla.org Graveyard
Statistics
P1
normal
RESOLVED FIXED
4 years ago
3 years ago

People

(Reporter: clouserw, Assigned: davidbgk)

Tracking

unspecified
2014-09
Dependency tree / graph

Details

(Whiteboard: [qa-])

Attachments

(2 attachments)

(Reporter)

Description

4 years ago
I didn't see an existing bug for this, but it's not a new problem.  

Lightweight Themes started as Personas on getpersonas.com, which means they had their own update URL that they would hit there.  When getpersonas.com was decommissioned the URL was 301'd to versioncheck.amo where it lives today.  This was done in bug 860815 which has a good summary of the history in comment 5.

This bug is to convert what we have today (a complicated pile of redirects) into something that works on HIVE (probably also a pile of redirects).  This includes:

* Numbers going into HIVE from our logs (ricardo)
* A query being written to get the numbers out of HIVE (ricardo)
* A script to pull those numbers and cache them in a file (AMO team)
* A script to parse that file and put the numbers in our db (note that this is simply the regular download and usage counts.  popularity is bug 1059964) (AMO team)

This task may be complicated a bit by 3 bugs that were spawned out of bug 860815 and are still open: bug 870835, bug 870838, bug 870841.  There is also bug 930491 which brings up the question of the redirects working as planned.  I bring these bugs up for completeness, not to widen scope - our goal is still to get equivalency with what we have today, and we can address these other bugs later.

Metrics bugs are hidden by default - if you need CCs hit me up on IRC.
(Reporter)

Updated

4 years ago
Blocks: 1059966
(Assignee)

Comment 1

4 years ago
> * A script to pull those numbers and cache them in a file (AMO team)

That step is currently done on Ricardo's team end for regular update/download counts as part of bug 1014752. We tried to do that on our side but it wasn't working as expected so Anurag generated the files to be able to push it into production before the end of July. Do we have to reconsider that process too? The issue was related to bandwidth/technical between prod servers and peach(?) if I remember correctly.
Assignee: nobody → david
(Assignee)

Updated

4 years ago
Depends on: 1061734
Just checked that the problem with bug 930491 is on. This is also a problem for getting LWT from Hive because some of the versionchecks (about 41%) are using the persona_id instead of addon_id.
The persona_id cannot be mapped to an addon_id using Hive. This needs to be done on AMO side so I'm exposing the "src" field in the Hive query. When the src=gp, then id will be the persona_id, if null it's the addon_id.

I've attached the Hive query to be used for the LWT and also a sample output if it's generated by Hive into a local file. The output is very small, 200K rows, 4MB total.

@David, you should try to run automate getting the data directly from Hive.
Created attachment 8483524 [details]
hive_lwt_query.sql
Created attachment 8483525 [details]
hive_lwt_output_sample.txt
First run failed:

Traceback (most recent call last):
  File "manage.py", line 130, in <module>
    execute_from_command_line()
  File "/data/mkt.prod/www/addons.mozilla.org/deploy-olympia-prod-20140903194725-4d1e688629/venv/lib/python2.6/site-packages/django/core/management/__init__.py", line 399, in execute_from_command_line
    utility.execute()
  File "/data/mkt.prod/www/addons.mozilla.org/deploy-olympia-prod-20140903194725-4d1e688629/venv/lib/python2.6/site-packages/django/core/management/__init__.py", line 392, in execute
    self.fetch_command(subcommand).run_from_argv(self.argv)
  File "/data/mkt.prod/www/addons.mozilla.org/deploy-olympia-prod-20140903194725-4d1e688629/venv/lib/python2.6/site-packages/django/core/management/base.py", line 242, in run_from_argv
    self.execute(*args, **options.__dict__)
  File "/data/mkt.prod/www/addons.mozilla.org/deploy-olympia-prod-20140903194725-4d1e688629/venv/lib/python2.6/site-packages/django/core/management/base.py", line 285, in execute
    output = self.handle(*args, **options)
  File "/data/mkt.prod/www/addons.mozilla.org/deploy-olympia-prod-20140903194725-4d1e688629/olympia/apps/stats/management/commands/__init__.py", line 44, in handle
    return query_to_file(self.query % (day, limit_str), filepath, sep)
  File "/data/mkt.prod/www/addons.mozilla.org/deploy-olympia-prod-20140903194725-4d1e688629/olympia/apps/stats/management/commands/hive_connection.py", line 56, in query_to_file
    with open(filepath, 'w') as f:
IOError: [Errno 13] Permission denied: '/mnt/netapp/mkt.prod/addons.mozilla.org/shared_storage/tmp/hive_results/2014-09-03/theme_update_counts.hive'


Jason, any idea?
Flags: needinfo?(jthomas)
[root@mktadm1 hive_results]# ls -l /mnt/netapp/mkt.prod/addons.mozilla.org/shared_storage/tmp/hive_results/2014-09-03/theme_update_counts.hive
ls: cannot access /mnt/netapp/mkt.prod/addons.mozilla.org/shared_storage/tmp/hive_results/2014-09-03/theme_update_counts.hive: No such file or directory

Is this file being pushed to us by the BI/DW team?
Flags: needinfo?(jthomas)
No sorry Jason, I should have been more clear: we're trying to create it in the previous cron, and it's failing (see comment 6). It seems it's failing because of "permission denied" (maybe the cron user doesn't have write access to this folder? If not, what should we do about it?)
I've adjusted the permissions to this directory.
(Assignee)

Updated

4 years ago
Status: NEW → RESOLVED
Last Resolved: 4 years ago
Resolution: --- → FIXED
Whiteboard: [qa-]
Product: addons.mozilla.org → addons.mozilla.org Graveyard
You need to log in before you can comment on or make changes to this bug.