910385 - ftp scraper not capturing b2g builds anymore

Status: VERIFIED WONTFIX

(Socorro :: Webapp, task)

Description

[Lonnen :lonnen]

For the last few weeks (how long?) the ftp scraper has been failing to capture b2g builds. Something changed on releng's side and we need to update to ftp scraper to get builds again.

Comment 1

[Robert Kaiser]

AFAIK, the internal for-device builds are not on public FTP and the end-user builds are not built by us and therefor in no place accessible to Mozilla, usually, both of which makes things harder for us here.

We should be able to scrape for the desktop builds though, so something is probably going wrong there.

Comment 2

[Robert Helmer [:rhelmer]]

(In reply to Robert Kaiser (:[e-mail]) from comment #1)
> AFAIK, the internal for-device builds are not on public FTP and the end-user
> builds are not built by us and therefor in no place accessible to Mozilla,
> usually, both of which makes things harder for us here.
> 
> We should be able to scrape for the desktop builds though, so something is
> probably going wrong there.

Yep, FTP directory changed slightly, here's a PR:
https://github.com/mozilla/socorro/pull/1451

Comment 3

[[github robot]]

Commits pushed to master at https://github.com/mozilla/socorro

https://github.com/mozilla/socorro/commit/12a59a3764e4dd70d9d35d71e3e5cfa21eabdf03
fixes bug 910385 - manifests have moved to nightly dir

https://github.com/mozilla/socorro/commit/1e9aaf47fd8f35a617e08dbb4edec6db5582050d
Merge pull request #1451 from rhelmer/bug910385-b2g-ftpscraper-broken

fixes bug 910385 - manifests have moved to nightly dir

Comment 4

[Matt Brandt [:mbrandt]]

Attachment: qa - reopened

Reopening for a bit of investigation - 

/builds/ on both stage and prod appear to be empty. Are we scraping/inserting into the db/presenting the data in the webapp correct?

https://crash-stats.mozilla.com/builds/products/B2G
https://crash-stats.allizom.org/builds/products/B2G

Comment 5

[Robert Helmer [:rhelmer]]

Did some preliminary investigation - looks like we are capturing the metadata into releases_raw table, so the basic premise of this bug is fixed. I think it's probably falling down in the normalization process somewhere, here's what I've found so far:

There are several "repository" entries which are not present in the "release_repositories" table - we surely do not want experimental branches, but we should investigate the ones like "mozilla-b2g18_v1_0_1" and "mozilla-b2g18_v1_0_0":

breakpad=> select distinct repository from releases_raw where product_name = 'b2g';
      repository      
----------------------
 oak
 mozilla-b2g18
 mozilla-central
 mozilla-b2g18_v1_0_1
 ionmonkey
 fx-team
 mozilla-aurora
 date
 mozilla-beta
 mozilla-b2g18_v1_0_0
 b2g-release
(11 rows)

Also I notice that the "build_type" column in releases_raw is "nightly" rather than "Nightly" for the latest b2g builds, not sure if this matters.. I'd suggest we match case-insensitively here (if we aren't already (from a quick grep through the stored procedures, I think this isn't the problem, just something I wanted to mention Just In Case)

Selena, would you mind looking at this? I suspect that we just need to add a few repositories to release_repositories and run backfill.

Comment 6

[Selena Deckelmann :selenamarie :selena]

Here are the repos that are reported in releases_raw but not in release_respositories: 

breakpad=# select distinct repository from releases_raw where product_name = 'b2g' AND repository NOT IN ( select repository from release_repositories); 
      repository      
----------------------
 oak
 mozilla-b2g18
 mozilla-b2g18_v1_0_1
 ionmonkey
 fx-team
 date
 mozilla-b2g18_v1_0_0
 b2g
(8 rows)


I'm attempting a backfill on stage to see what happens...

Comment 7

[Selena Deckelmann :selenamarie :selena]

I think the relevant code is: https://github.com/mozilla/socorro/blob/master/socorro/external/postgresql/products_builds.py#L45

Maybe tomorrow someone can help me set up local middleware so I can get it to spit out the SQL. I think there's a problem with the date logic.

Comment 8

[Robert Helmer [:rhelmer]]

(In reply to Selena Deckelmann :selenamarie :selena from comment #7)
> I think the relevant code is:
> https://github.com/mozilla/socorro/blob/master/socorro/external/postgresql/
> products_builds.py#L45
> 
> Maybe tomorrow someone can help me set up local middleware so I can get it
> to spit out the SQL. I think there's a problem with the date logic.

Using local mware is documented on mana now - I know both peterbe and I have it going ok (I use it quite a bit) so I am sure one of us can help.

The easiest way to get it to spit out SQL is to uncomment:
https://github.com/mozilla/socorro/blob/master/socorro/external/postgresql/base.py#L86

I noticed there's a typo on that line though and have neglected to file a bug, it should be:
  self.context.logger.debug(connection.cursor().mogrify(sql, params))
( n.b. cursor() not cursor )

Comment 9

[Selena Deckelmann :selenamarie :selena]

Is this still an issue?

Comment 10

[Lonnen :lonnen]

no longer a concern

Comment 11

[Matt Brandt [:mbrandt]]

Bumping to verified wontfix.