832083 - add support for b2g-style json manifest files to ftpscraper

Status: VERIFIED FIXED

(Socorro :: Backend, task)

Description

[Robert Helmer [:rhelmer]]

Needed for bug 821989

Comment 1

[Robert Helmer [:rhelmer]]

Examples can be found in:
http://ftp.mozilla.org/pub/mozilla.org/b2g/manifests/

For instance:
http://ftp.mozilla.org/pub/mozilla.org/b2g/manifests/latest/source_unagi-stable_2013-01-18.xml

Comment 2

[Selena Deckelmann :selenamarie :selena]

New file format example: 

http://ftp.mozilla.org/pub/mozilla.org/b2g/manifests/2013-01-25-07/socorro_unagi_2013-01-25-07.json

{"buildid": "20130125070201", "update_channel": "nightly", "version": "18.0"}

Comment 3

[[github robot]]

Commits pushed to master at https://github.com/mozilla/socorro

https://github.com/mozilla/socorro/commit/65e546ff4ff3d23ebe2d204e30f509551a9c7c94
bug 832083 adding B2G to our scraper

https://github.com/mozilla/socorro/commit/35dbec09008323647ef82dbb3ec0583885eabdfe
Merge pull request #1067 from selenamarie/bug832083-add-b2g-builds

bug 832083 adding B2G to our scraper

Comment 4

[Matt Brandt [:mbrandt]]

rhelmer: can you suggest minimal steps for QA to test this or is this [qa-] until bug 821989 lands?

Comment 5

[Selena Deckelmann :selenamarie :selena]

Sample of recent inserts into the releases_raw table: 

psql> select * from releases_raw order by build_id desc limit 10;

-[ RECORD 1 ]+-----------------------
product_name | b2g
version      | 18.0
platform     | mac64-localizer
build_id     | 20130227070202
build_type   | Nightly
beta_number  | 
repository   | mozilla-b2g18
-[ RECORD 2 ]+-----------------------
product_name | b2g
version      | 18.0
platform     | win32-localizer
build_id     | 20130227070202
build_type   | Nightly
beta_number  | 
repository   | mozilla-b2g18

Comment 6

[Matt Brandt [:mbrandt]]

Attachment: qa - verified on stage

QA verifed on stage -  thank you Selena for helping unravel the thread on where to look.

https://crash-stats.allizom.org/products/B2G/builds

Comment 7

[Robert Helmer [:rhelmer]]

This was not ported to crontabber FTPScraper, need that for rapid beta support for B2G :)

Comment 8

[Selena Deckelmann :selenamarie :selena]

r? https://github.com/mozilla/socorro/pull/1178

Comment 9

[Selena Deckelmann :selenamarie :selena]

Work on this is underway.

Comment 10

[[github robot]]

Commits pushed to master at https://github.com/mozilla/socorro

https://github.com/mozilla/socorro/commit/c6fabc99983318da960b18cb5ddb3d72d56d76a8
bug 832083 - update ftpscraper b2g code to support crontabber

Thanks, @peterbe for a great review!

https://github.com/mozilla/socorro/commit/8f1ea30c6c283de59fb0eea5b593fa93e8c2d628
Merge pull request #1178 from selenamarie/832083-scraper-to-crontabber

bug 832083 - update ftpscraper b2g code to support crontabber

Comment 11

[Matt Brandt [:mbrandt]]

Attachment: qa - reopened

Reopened:

I don't see entries for the latest nightly builds:
- https://crash-stats.allizom.org/products/Firefox/builds
- https://crash-stats.allizom.org/products/B2G/builds

Comment 12

[Robert Helmer [:rhelmer]]

Going to follow up on this.

Comment 13

[Robert Helmer [:rhelmer]]

This seems to be working fine, I think the current problem is further along in the process (see bug 864896)

Comment 14

[Matt Brandt [:mbrandt]]

(In reply to Robert Helmer [:rhelmer] from comment #13)
> This seems to be working fine, I think the current problem is further along
> in the process (see bug 864896)

rhelmer, thanks for investigating and opening bug 864896 for the B2G/builds.

QA verified https://crash-stats.allizom.org/products/Firefox/builds is now populated.