ftpscraper fails with IndexError on mobile

RESOLVED FIXED

Status

Socorro
General
RESOLVED FIXED
2 years ago
2 years ago

People

(Reporter: peterbe, Unassigned)

Tracking

Firefox Tracking Flags

(Not tracked)

Details

(Reporter)

Description

2 years ago
See https://sentry.prod.mozaws.net/operations/socorro-prod/group/237348/

I can reproduce this locally by running: 

  python socorro/cron/jobs/ftpscraper.py --product=mobile --date=2016-02-21

Here's the error:

Traceback (most recent call last):
  File "socorro/cron/jobs/ftpscraper.py", line 731, in <module>
    sys.exit(main(FTPScraperCronAppDryRunner))
  File "/Users/peterbe/dev/MOZILLA/SOCORRO/socorro/socorro/app/socorro_app.py", line 506, in main
    values_source_list=values_source_list
  File "/Users/peterbe/dev/MOZILLA/SOCORRO/socorro/socorro/app/socorro_app.py", line 191, in run
    values_source_list=values_source_list
  File "/Users/peterbe/dev/MOZILLA/SOCORRO/socorro/socorro/app/socorro_app.py", line 289, in _do_run
    return_code = fix_exit_code(app_to_run.main())
  File "socorro/cron/jobs/ftpscraper.py", line 727, in main
    self.ftpscraper.run(self.config.date)
  File "/Users/peterbe/dev/MOZILLA/SOCORRO/socorro/socorro/cron/jobs/ftpscraper.py", line 383, in run
    date
  File "/Users/peterbe/virtualenvs/socorro/lib/python2.7/site-packages/crontabber/transaction_executor.py", line 46, in __call__
    result = function(connection, *args, **kwargs)
  File "/Users/peterbe/dev/MOZILLA/SOCORRO/socorro/socorro/cron/jobs/ftpscraper.py", line 388, in _scrape_releases_and_nightlies
    self.scrape_nightlies(connection, product_name, date)
  File "/Users/peterbe/dev/MOZILLA/SOCORRO/socorro/socorro/cron/jobs/ftpscraper.py", line 611, in scrape_nightlies
    for info in self.get_nightly(nightly, dirname):
  File "/Users/peterbe/dev/MOZILLA/SOCORRO/socorro/socorro/cron/jobs/ftpscraper.py", line 295, in get_nightly
    kvpairs, bad_lines = self.parse_info_file(url, nightly=True)
  File "/Users/peterbe/dev/MOZILLA/SOCORRO/socorro/socorro/cron/jobs/ftpscraper.py", line 146, in parse_info_file
    results = {'buildID': contents[0], 'rev': contents[1]}
IndexError: list index out of range
(Reporter)

Updated

2 years ago
Depends on: 1249993
(Reporter)

Comment 1

2 years ago
Kairo, 
As you can see, our ftpscraper is repeatedly stuck and it's holding up crontabber. The problem seems to be that releng has put a non-nightly _info.txt file under a nightly release. 
I can reproduce this locally so it wouldn't be particularly hard to hack around this. But I'm not sure if we should. Hoping for some feedback from you and Rail.

Comment 2

2 years ago
I prefer the FTP scraper working and just ignoring a file it can't deal with so at least the rest would work. For example, I need Firefox 45 beta 8 to be picked up Monday morning, otherwise the already-in-jeopardy Firefox 45 release will be in even more trouble than it already is.
(Reporter)

Updated

2 years ago
Depends on: 1250142
What's the status here ? If it's still a problem it would be good to get some urls which are causing problems.
(Reporter)

Comment 4

2 years ago
The reason there were two bugs was

1) One to track changing the code over so it starts reading the .json files instead of _info.txt

2) Being impatient and waiting for 1) let's fix what's broken "right now". 

I think all is well now. We're happily reading those .json files for mobile in prod now and have for some time.
Status: NEW → RESOLVED
Last Resolved: 2 years ago
Resolution: --- → FIXED
You need to log in before you can comment on or make changes to this bug.