660859 - input.mozilla.com cronjob to export opinions seems to be stuck

Status: VERIFIED FIXED

(Input Graveyard :: Backend, defect)

Description

[Michael Kurze [:michaelk]]

The opinions export cron job has not completed since April 27th.

It seems to be stuck somewhere in the middle, the output file remains at the "exporting" stage

We probably need an excerpt from the cron logs to determine what is wrong here. Maybe a file needs to be deleted or an old process needs to be killed.

You can ping wenzel or me (michaelk) in #input. Thanks!

Comment 1

[Jake Maul [:jakem]]

Do you have any more information on this? I don't know what cron job is responsible for this operation.

on pp-app-input01 and 02, I see: /data/www/django/input.mozilla.com/tsv_exports

But the corresponding directory does not exist on the admin hosts, mradm02 and ip-admin02.


Any additional info you can provide as to how this directory (and the files within) get created would be helpful. Thanks!

Comment 2

[Jake Maul [:jakem]]

wenzel pointed me in the right direction, from a note in a previous bug. The cron job actually runs on pp-app-input02, and the files are stored on an NFS mount. It doesn't run on mradm02 anymore because the NFS mount is on pio-netapp-a, in Phoenix.

I touched a file in there to give a hint to anyone else investigating this in the future.

Once the cron job was located, wenzel had me point it to [e-mail], so in the future they'll get emails on it.

I ran it once by hand. It does fail, some time between 1190000 and 1200000 records in:

1170000
1180000
1190000
Traceback (most recent call last):
  File "/data/www/django/input.mozilla.com/reporter/manage.py", line 56, in <module>
    execute_manager(settings)
  File "/data/www/django/input.mozilla.com/reporter/vendor/packages/Django/django/core/management/__init__.py", line 438, in execute_manager
    utility.execute()
  File "/data/www/django/input.mozilla.com/reporter/vendor/packages/Django/django/core/management/__init__.py", line 379, in execute
    self.fetch_command(subcommand).run_from_argv(self.argv)
  File "/data/www/django/input.mozilla.com/reporter/vendor/packages/Django/django/core/management/base.py", line 191, in run_from_argv
    self.execute(*args, **options.__dict__)
  File "/data/www/django/input.mozilla.com/reporter/vendor/packages/Django/django/core/management/base.py", line 220, in execute
    output = self.handle(*args, **options)
  File "/data/www/django/input.mozilla.com/reporter/vendor/src/django-cronjobs/cronjobs/management/commands/cron.py", line 38, in handle
    registered[script](*args)
  File "/data/www/django/input.mozilla.com/reporter/apps/api/cron.py", line 71, in export_tsv
    OPINION_TYPES.get(opinion.type).short,
AttributeError: 'NoneType' object has no attribute 'short'


I think this is probably out of IT's hands now, so I'm assigning it to wenzel for further investigation / delegation.

Comment 3

[Fred Wenzel [:wenzel]]

This is live now. Michael, can you check tomorrow to see if the files updated as they should?

Comment 4

[Michael Kurze [:michaelk]]

To quote the latest export:

2104661	1307005138	praise	firefox	6.0a2	winxp	en-US				BEST BROWSER EVER :)\
SUPER FAST.


-> 1307005138 ^= Thu, 02 Jun 2011 08:58:58 GMT
-> VERIFIED FIXED

Comment 5

[Fred Wenzel [:wenzel]]

Great! Thanks, Jake and Mike!