931803 - releases_raw doesn't have data after 10-17 (ftpscraper bustage?)

Status: VERIFIED FIXED

(Socorro :: General, task)

Description

[Benjamin Smedberg]

The releases_raw table, which I believe is populated from ftpscraper, doesn't have nightly builds after 10-17. This makes it hard or impossible to do crashes-per-ADU-per-buildID numbers for bug verifications.

<bsmedberg> KaiRo/selenamarie: aha, the ADU tables are ok, but releases_raw doesn't have the builds after 10-17
<bsmedberg> does that indicate an ftpscraper issue?
<selenamarie> bsmedberg: yup -- i see that ftpscraper has been busted for a couple days as well
<bsmedberg> selenamarie: bug filed or are you on it?
<selenamarie> bsmedberg: i haven't seen any bugs, but would appreciate one or more :)
* bsmedberg will file
<selenamarie> oh awesome: ftpscraper error is --> HTTPError: HTTP Error 404: Not Found
<selenamarie> lol
<selenamarie> sigh

This is urgent for me to do some nightly verifications near today's uplift.

Comment 1

[Selena Deckelmann :selenamarie :selena]

Command is: 

(socorro-virtualenv)[[e-mail]1 application]$ PYTHONPATH=. ../socorro-virtualenv/bin/python socorro/cron/crontabber.py --admin.conf=/etc/socorro/crontabber.ini -j ftpscraper

Confirmed that the error is coming from scraping b2g.

Comment 2

[Peter Bengtsson [:peterbe]]

Attachment: Github PR

Comment 3

[Selena Deckelmann :selenamarie :selena]

We should be updated now for Firefox: 

PYTHONPATH=. ../socorro-virtualenv/bin/python socorro/cron/crontabber.py --admin.conf=/etc/socorro/crontabber.ini -j ftpscraper --crontabber.class-FTPScraperCronApp.products=firefox

Most recent nightly in releases_raw is: 20131028040205 | firefox

And I see releases from the 17th through today.

b2g is still broken, with some xml errors. Looking into it with :peterbe.

Comment 4

[Selena Deckelmann :selenamarie :selena]

Finished running backfill for ftpscraper from 10/17->today. 

:bsmedberg -- please have a look and let me know if we are still missing releases.

Comment 5

[Benjamin Smedberg]

I now see at least the Windows nightly builds that I expect to see. I didn't examine everything carefully, but I don't see obvious problems.

Comment 6

[[github robot]]

Commit pushed to master at https://github.com/mozilla/socorro

https://github.com/mozilla/socorro/commit/9e8bf28f4996c28326dd60bf9529150d91015f36
bug 931803 - ftpscraper to ignore 404 errors, r=selenamarie

Comment 7

[Matt Brandt [:mbrandt]]

Bumping to verified as [qa-].