864018 - "Nightly Builds" entries missing from UI since April 19

Status: RESOLVED FIXED

(Socorro :: General, task)

Description

[Scoobidiver (away)]

The latest builds don't show up in https://crash-stats.mozilla.com/products/Firefox/builds

Comment 1

[Robert Kaiser]

Rob, is this a Socorro 42 FTP scraper regression or only a hiccup of some sort?

Comment 2

[Robert Helmer [:rhelmer]]

(In reply to Scoobidiver from comment #0)
> The latest builds don't show up in
> https://crash-stats.mozilla.com/products/Firefox/builds

(In reply to Robert Kaiser (:[e-mail]) from comment #1)
> Rob, is this a Socorro 42 FTP scraper regression or only a hiccup of some
> sort?

The timing coincides with the release of 42, lonnen I believe you were talking in IRC about backing out the FTPScraper changes since it didn't make the warnings go away - did you end up doing that?

We should probably back this out if not, since it'll affect our ability to pick up new releases.

Comment 3

[Lonnen :lonnen]

The morning of the 42 release --

* I noticed the stage FTP scraper was erroring every time it ran (b2g scraping, iirc)
* Peter mentioned Selena had written fixes but they were on dev only
* I cherry picked two fixes to stage, one trivial
* Stage continued to error every hour (also, b2g scraping)
* QA did not feel comfortable with a non-trivial FTPScraper change an hour before the release that didn't fix our problem
* We backed out the non-trivial change
* Release 42

There may be a regression in the 42 FTPScraper code, or it may be failing hard during B2G, causing it to skip whatever scraping work it has left. I don't think ftpscraper has ran successfully since our last release.

Comment 4

[Lonnen :lonnen]

FWIW, the FTPScraper is failing to handle a 404.

2013-04-21 11:14:13,906 DEBUG - MainThread - error when running <class 'socorro.cron.jobs.ftpscraper.FTPScraperCronApp'> on None
Traceback (most recent call last):
  File "/data/socorro/application/socorro/cron/crontabber.py", line 701, in _run_one
    for last_success in self._run_job(job_class, config, info):
  File "/data/socorro/application/socorro/cron/base.py", line 173, in main
    function(when)
  File "/data/socorro/application/socorro/cron/base.py", line 212, in _run_proxy
    self.run(connection, date)
  File "/data/socorro/application/socorro/cron/jobs/ftpscraper.py", line 192, in run
    self.scrapeB2G(connection, product_name, date)
  File "/data/socorro/application/socorro/cron/jobs/ftpscraper.py", line 283, in scrapeB2G
    nightlies = getLinks(prod_url, startswith=dir_prefix)
  File "/data/socorro/application/socorro/cron/jobs/ftpscraper.py", line 32, in getLinks
    page = urllib2.urlopen(url)
  File "/usr/lib64/python2.6/urllib2.py", line 126, in urlopen
    return _opener.open(url, data, timeout)
  File "/usr/lib64/python2.6/urllib2.py", line 397, in open
    response = meth(req, response)
  File "/usr/lib64/python2.6/urllib2.py", line 510, in http_response
    'http', request, response, code, msg, hdrs)
  File "/usr/lib64/python2.6/urllib2.py", line 435, in error
    return self._call_chain(*args)
  File "/usr/lib64/python2.6/urllib2.py", line 369, in _call_chain
    result = func(*args)
  File "/usr/lib64/python2.6/urllib2.py", line 518, in http_error_default
    raise HTTPError(req.get_full_url(), code, msg, hdrs, fp)
HTTPError: HTTP Error 404: Not Found

Comment 5

[Selena Deckelmann :selenamarie :selena]

(In reply to Chris Lonnen :lonnen from comment #4)
> FWIW, the FTPScraper is failing to handle a 404.
...
> HTTPError: HTTP Error 404: Not Found

Sorry :/

I just had a look and it appears that stage is running ok now. 

Was there another bug to fix my commits?

Comment 6

[Selena Deckelmann :selenamarie :selena]

Just to confirm from postgres - the latest B2G builds are present: 

 product_name | version | platform  |    build_id    | build_type | beta_number 
| repository  
--------------+---------+-----------+----------------+------------+-------------
+-------------
 b2g          | 18.0    | hamachi   | 20130421070204 | nightly    |             
| b2g-release
 b2g          | 18.0    | unagi     | 20130421070204 | nightly    |             
| b2g-release
 b2g          | 18.0    | unagi-eng | 20130421070204 | nightly    |             
| b2g-release
 b2g          | 18.0    | otoro     | 20130421070204 | nightly    |             
| b2g-release
 b2g          | 18.0    | inari     | 20130421070204 | nightly    |             
| b2g-release
 b2g          | 18.0    | inari     | 20130421070203 | nightly    |             
| b2g-release
...
 b2g          | 18.0    | unagi    | 20130421070203 | beta       |           1 |
 b2g-release
 b2g          | 18.0    | unagi    | 20130420070203 | beta       |           1 |
 b2g-release
 b2g          | 18.0    | unagi    | 20130419070205 | beta       |           1 |
 b2g-release
 seamonkey    | 2.18    | macosx64 | 20130418194903 | Beta       |           2 |
 mozilla-beta


-- including Beta which were broken before the v43 commits to dev.

Comment 7

[Robert Helmer [:rhelmer]]

(In reply to Selena Deckelmann :selenamarie :selena from comment #6)
> Just to confirm from postgres - the latest B2G builds are present: 

How does Firefox look (see link in comment 0)?

Comment 8

[Robert Helmer [:rhelmer]]

(In reply to Robert Helmer [:rhelmer] from comment #7)
> (In reply to Selena Deckelmann :selenamarie :selena from comment #6)
> > Just to confirm from postgres - the latest B2G builds are present: 
> 
> How does Firefox look (see link in comment 0)?

Also builds are not showing up at https://crash-stats.mozilla.com/products/B2G/builds so something is wrong, I believe there's a matview for that.

Comment 9

[Selena Deckelmann :selenamarie :selena]

(In reply to Robert Helmer [:rhelmer] from comment #8)
> (In reply to Robert Helmer [:rhelmer] from comment #7)
> > (In reply to Selena Deckelmann :selenamarie :selena from comment #6)
> > > Just to confirm from postgres - the latest B2G builds are present: 
> > 
> > How does Firefox look (see link in comment 0)?
> 
> Also builds are not showing up at
> https://crash-stats.mozilla.com/products/B2G/builds so something is wrong, I
> believe there's a matview for that.

Yes, sure. What I'm saying is that the scraper is working in *stage*.  :)  From comment #3, the team decided not deploy the cherry-picked changes from v43 that are currently working properly on stage. I was documenting that, for whatever reason, my patches *appear* to be working on stage. 

The data and patches I am referring to *are not* on prod yet.

Here is more stage data for firefox builds (I also checked beta): 

 firefox      | 23.0a1  | linux-x86_64       | 20130421031002 | Nightly    |    
         | mozilla-central
 firefox      | 23.0a1  | mac                | 20130421031002 | Nightly    |    
         | mozilla-central
 firefox      | 23.0a1  | win32              | 20130421031002 | Nightly    |    
         | mozilla-central
 firefox      | 23.0a1  | linux-x86_64-asan  | 20130421031002 | Nightly    |    
         | mozilla-central
 firefox      | 23.0a1  | linux-i686         | 20130421031002 | Nightly    |    
         | mozilla-central
 firefox      | 23.0a1  | win64-x86_64       | 20130421031002 | Nightly    |    
         | mozilla-central

Comment 10

[Robert Helmer [:rhelmer]]

(In reply to Selena Deckelmann :selenamarie :selena from comment #9)
> (In reply to Robert Helmer [:rhelmer] from comment #8)
> > (In reply to Robert Helmer [:rhelmer] from comment #7)
> > > (In reply to Selena Deckelmann :selenamarie :selena from comment #6)
> > > > Just to confirm from postgres - the latest B2G builds are present: 
> > > 
> > > How does Firefox look (see link in comment 0)?
> > 
> > Also builds are not showing up at
> > https://crash-stats.mozilla.com/products/B2G/builds so something is wrong, I
> > believe there's a matview for that.
> 
> Yes, sure. What I'm saying is that the scraper is working in *stage*.  :) 
> From comment #3, the team decided not deploy the cherry-picked changes from
> v43 that are currently working properly on stage. I was documenting that,
> for whatever reason, my patches *appear* to be working on stage. 
> 
> The data and patches I am referring to *are not* on prod yet.


Ah ok sorry :) So should we deploy the changes then, since they are currently working on stage? Or try to figure out what went wrong?

There's probably less risk in just deploying the changes and then figuring out what went wrong, since it's currently broken anyway.

Comment 11

[Lonnen :lonnen]

I won't have time for a release until 3:30 PST today, 4-22.

We should consider just pushing 43 early. Lot's of cherry picks last week makes it hard to tell at a glance, but the only new things staged for this week are Selena's FTPScraper fix and `bug 855716 - safely handle tmpdir and use lock correctly`.

Comment 12

[Scoobidiver (away)]

It seems to be fixed.