bugzilla.mozilla.org has resumed normal operation. Attachments prior to 2014 will be unavailable for a few days. This is tracked in Bug 1475801.
Please report any other irregularities here.
Please report any other irregularities here.
I didn't see an existing bug for this, but it's not a new problem. Lightweight Themes started as Personas on getpersonas.com, which means they had their own update URL that they would hit there. When getpersonas.com was decommissioned the URL was 301'd to versioncheck.amo where it lives today. This was done in bug 860815 which has a good summary of the history in comment 5. This bug is to convert what we have today (a complicated pile of redirects) into something that works on HIVE (probably also a pile of redirects). This includes: * Numbers going into HIVE from our logs (ricardo) * A query being written to get the numbers out of HIVE (ricardo) * A script to pull those numbers and cache them in a file (AMO team) * A script to parse that file and put the numbers in our db (note that this is simply the regular download and usage counts. popularity is bug 1059964) (AMO team) This task may be complicated a bit by 3 bugs that were spawned out of bug 860815 and are still open: bug 870835, bug 870838, bug 870841. There is also bug 930491 which brings up the question of the redirects working as planned. I bring these bugs up for completeness, not to widen scope - our goal is still to get equivalency with what we have today, and we can address these other bugs later. Metrics bugs are hidden by default - if you need CCs hit me up on IRC.
> * A script to pull those numbers and cache them in a file (AMO team) That step is currently done on Ricardo's team end for regular update/download counts as part of bug 1014752. We tried to do that on our side but it wasn't working as expected so Anurag generated the files to be able to push it into production before the end of July. Do we have to reconsider that process too? The issue was related to bandwidth/technical between prod servers and peach(?) if I remember correctly.
Assignee: nobody → david
Just checked that the problem with bug 930491 is on. This is also a problem for getting LWT from Hive because some of the versionchecks (about 41%) are using the persona_id instead of addon_id. The persona_id cannot be mapped to an addon_id using Hive. This needs to be done on AMO side so I'm exposing the "src" field in the Hive query. When the src=gp, then id will be the persona_id, if null it's the addon_id. I've attached the Hive query to be used for the LWT and also a sample output if it's generated by Hive into a local file. The output is very small, 200K rows, 4MB total. @David, you should try to run automate getting the data directly from Hive.
Fix pushed in https://github.com/mozilla/olympia/commit/3e8497d73b6f0036984d7b7ed154233f20086ffa Let's see what results this gives.
First run failed: Traceback (most recent call last): File "manage.py", line 130, in <module> execute_from_command_line() File "/data/mkt.prod/www/addons.mozilla.org/deploy-olympia-prod-20140903194725-4d1e688629/venv/lib/python2.6/site-packages/django/core/management/__init__.py", line 399, in execute_from_command_line utility.execute() File "/data/mkt.prod/www/addons.mozilla.org/deploy-olympia-prod-20140903194725-4d1e688629/venv/lib/python2.6/site-packages/django/core/management/__init__.py", line 392, in execute self.fetch_command(subcommand).run_from_argv(self.argv) File "/data/mkt.prod/www/addons.mozilla.org/deploy-olympia-prod-20140903194725-4d1e688629/venv/lib/python2.6/site-packages/django/core/management/base.py", line 242, in run_from_argv self.execute(*args, **options.__dict__) File "/data/mkt.prod/www/addons.mozilla.org/deploy-olympia-prod-20140903194725-4d1e688629/venv/lib/python2.6/site-packages/django/core/management/base.py", line 285, in execute output = self.handle(*args, **options) File "/data/mkt.prod/www/addons.mozilla.org/deploy-olympia-prod-20140903194725-4d1e688629/olympia/apps/stats/management/commands/__init__.py", line 44, in handle return query_to_file(self.query % (day, limit_str), filepath, sep) File "/data/mkt.prod/www/addons.mozilla.org/deploy-olympia-prod-20140903194725-4d1e688629/olympia/apps/stats/management/commands/hive_connection.py", line 56, in query_to_file with open(filepath, 'w') as f: IOError: [Errno 13] Permission denied: '/mnt/netapp/mkt.prod/addons.mozilla.org/shared_storage/tmp/hive_results/2014-09-03/theme_update_counts.hive' Jason, any idea?
[root@mktadm1 hive_results]# ls -l /mnt/netapp/mkt.prod/addons.mozilla.org/shared_storage/tmp/hive_results/2014-09-03/theme_update_counts.hive ls: cannot access /mnt/netapp/mkt.prod/addons.mozilla.org/shared_storage/tmp/hive_results/2014-09-03/theme_update_counts.hive: No such file or directory Is this file being pushed to us by the BI/DW team?
No sorry Jason, I should have been more clear: we're trying to create it in the previous cron, and it's failing (see comment 6). It seems it's failing because of "permission denied" (maybe the cron user doesn't have write access to this folder? If not, what should we do about it?)
I've adjusted the permissions to this directory.
Status: NEW → RESOLVED
Last Resolved: 4 years ago
Resolution: --- → FIXED
Product: addons.mozilla.org → addons.mozilla.org Graveyard
You need to log in before you can comment on or make changes to this bug.