Closed Bug 1249992 Opened 8 years ago Closed 8 years ago

ftpscraper fails with IndexError on mobile

Categories

(Socorro :: General, task)

task
Not set
normal

Tracking

(Not tracked)

RESOLVED FIXED

People

(Reporter: peterbe, Unassigned)

References

Details

See https://sentry.prod.mozaws.net/operations/socorro-prod/group/237348/

I can reproduce this locally by running: 

  python socorro/cron/jobs/ftpscraper.py --product=mobile --date=2016-02-21

Here's the error:

Traceback (most recent call last):
  File "socorro/cron/jobs/ftpscraper.py", line 731, in <module>
    sys.exit(main(FTPScraperCronAppDryRunner))
  File "/Users/peterbe/dev/MOZILLA/SOCORRO/socorro/socorro/app/socorro_app.py", line 506, in main
    values_source_list=values_source_list
  File "/Users/peterbe/dev/MOZILLA/SOCORRO/socorro/socorro/app/socorro_app.py", line 191, in run
    values_source_list=values_source_list
  File "/Users/peterbe/dev/MOZILLA/SOCORRO/socorro/socorro/app/socorro_app.py", line 289, in _do_run
    return_code = fix_exit_code(app_to_run.main())
  File "socorro/cron/jobs/ftpscraper.py", line 727, in main
    self.ftpscraper.run(self.config.date)
  File "/Users/peterbe/dev/MOZILLA/SOCORRO/socorro/socorro/cron/jobs/ftpscraper.py", line 383, in run
    date
  File "/Users/peterbe/virtualenvs/socorro/lib/python2.7/site-packages/crontabber/transaction_executor.py", line 46, in __call__
    result = function(connection, *args, **kwargs)
  File "/Users/peterbe/dev/MOZILLA/SOCORRO/socorro/socorro/cron/jobs/ftpscraper.py", line 388, in _scrape_releases_and_nightlies
    self.scrape_nightlies(connection, product_name, date)
  File "/Users/peterbe/dev/MOZILLA/SOCORRO/socorro/socorro/cron/jobs/ftpscraper.py", line 611, in scrape_nightlies
    for info in self.get_nightly(nightly, dirname):
  File "/Users/peterbe/dev/MOZILLA/SOCORRO/socorro/socorro/cron/jobs/ftpscraper.py", line 295, in get_nightly
    kvpairs, bad_lines = self.parse_info_file(url, nightly=True)
  File "/Users/peterbe/dev/MOZILLA/SOCORRO/socorro/socorro/cron/jobs/ftpscraper.py", line 146, in parse_info_file
    results = {'buildID': contents[0], 'rev': contents[1]}
IndexError: list index out of range
Depends on: 1249993
Kairo, 
As you can see, our ftpscraper is repeatedly stuck and it's holding up crontabber. The problem seems to be that releng has put a non-nightly _info.txt file under a nightly release. 
I can reproduce this locally so it wouldn't be particularly hard to hack around this. But I'm not sure if we should. Hoping for some feedback from you and Rail.
I prefer the FTP scraper working and just ignoring a file it can't deal with so at least the rest would work. For example, I need Firefox 45 beta 8 to be picked up Monday morning, otherwise the already-in-jeopardy Firefox 45 release will be in even more trouble than it already is.
Depends on: 1250142
What's the status here ? If it's still a problem it would be good to get some urls which are causing problems.
The reason there were two bugs was

1) One to track changing the code over so it starts reading the .json files instead of _info.txt

2) Being impatient and waiting for 1) let's fix what's broken "right now". 

I think all is well now. We're happily reading those .json files for mobile in prod now and have for some time.
Status: NEW → RESOLVED
Closed: 8 years ago
Resolution: --- → FIXED
You need to log in before you can comment on or make changes to this bug.