679541 - enable mobile in build cron (ftp scraper) config

Status: VERIFIED FIXED

(Socorro :: General, task)

Description

[Robert Helmer [:rhelmer]]

Now that bug 675272 is fixed, we should:

1) enable mobile in buildsconfig.py.dist (add "mobile/candidates" 

This requires adding "mobile/candidates/" to "release_product_uris.default" 

2) run on stage, make sure release_raw is populated correctly 

The script should automatically backfill any entries that are needed.

3) deploy above config change to prod

Then we can disable the release emails to socorro-dev (yay!)

Comment 1

[Robert Helmer [:rhelmer]]

Attachment: scan mobile/candidates dir

Now that bug 675272 is fixed, we should scan the mobile/candidates directory (unlike the older products, it has a dedicated separate dir instead of reusing "nightly").

I've tested this locally with a mirror I made. We already have the needed platforms in release_platforms.

Comment 2

[Lonnen :lonnen]

Comment on attachment 555762 [details] [diff] [review]
scan mobile/candidates dir

I don't have a local test mirror, but this change to our code base looks fine.

Comment 3

[Robert Helmer [:rhelmer]]

Trunk:
Committed revision 3476.
2.2 Branch:
Committed revision 3477.

Comment 4

[Robert Helmer [:rhelmer]]

OK so this was enabled on stage, and does work (from admin1:/var/log/socorro/cron_builds.log):

2011-08-30 01:30:01,801 INFO -  release_product_uris=('firefox/nightly/', 'seamonkey/nightly/', 'thunderbird/nightly/', 'mobile/candidates/')
(...)
2011-08-30 01:30:47,054 INFO - Did not find build entries in releases_raw table for mobile 4.0rc1 20110318105707 Release maemo5-gtk None
2011-08-30 01:30:47,055 INFO - Inserted the following build: mobile 4.0rc1 maemo5-gtk 20110318105707 Release None

However, I see from looking in the releases_raw table that it's not finding any of the new builds that should be picked up since bug 675272 was fixed.

I think the reason is that the .txt file names (which we try to match against release_platforms) are different than what we expected/what mobile used in the past:

Compare:
http://stage.mozilla.org/pub/mozilla.org/mobile/candidates/7.0b3-candidates/build1/

With:
2011-08-30 07:05:01,762 INFO -  release_platforms=('linux', 'linux64', 'macosx64', 'macosx', 'win32', 'android', 'android-r7', 'linux-i686', 'macosx-i686', 'maemo5-gtk', 'win32-i686')

I think we should just add the new platform names and be done with it.

Comment 5

[Robert Helmer [:rhelmer]]

Attachment: add new mobile platform names

With this patch, seems to work fine for me. 

2011-08-30 13:32:36,481 INFO - Did not find build entries in releases_raw table for mobile 6.0.1 20110830102437 Release linux-android None
2011-08-30 13:32:36,484 INFO - Inserted the following build: mobile 6.0.1 linux-android 20110830102437 Release None
2011-08-30 13:32:36,493 INFO - Did not find build entries in releases_raw table for mobile 6.0.1 20110830100432 Release linux-maemo5-gtk None
2011-08-30 13:32:36,495 INFO - Inserted the following build: mobile 6.0.1 linux-maemo5-gtk 20110830100432 Release None
2011-08-30 13:32:36,502 INFO - Did not find build entries in releases_raw table for mobile 6.0.1 20110830100141 Release linux-mobile None
2011-08-30 13:32:36,504 INFO - Inserted the following build: mobile 6.0.1 linux-mobile [phone] Release None
2011-08-30 13:32:36,509 INFO - Did not find build entries in releases_raw table for mobile 6.0.1 20110830110357 Release macosx-mobile None
2011-08-30 13:32:36,511 INFO - Inserted the following build: mobile 6.0.1 macosx-mobile [phone] Release None
2011-08-30 13:32:36,516 INFO - Did not find build entries in releases_raw table for mobile 6.0.1 20110830104432 Release win32-mobile None
2011-08-30 13:32:36,518 INFO - Inserted the following build: mobile 6.0.1 win32-mobile [phone] Release None
2011-08-30 13:32:37,308 INFO - Did not find build entries in releases_raw table for mobile 7.0 20110816154147 Beta linux-android 1
2011-08-30 13:32:37,312 INFO - Inserted the following build: mobile 7.0 linux-android 20110816154147 Beta 1
2011-08-30 13:32:37,317 INFO - Did not find build entries in releases_raw table for mobile 7.0 20110816161304 Beta linux-maemo5-gtk 1
2011-08-30 13:32:37,319 INFO - Inserted the following build: mobile 7.0 linux-maemo5-gtk 20110816161304 Beta 1
2011-08-30 13:32:37,325 INFO - Did not find build entries in releases_raw table for mobile 7.0 20110816160908 Beta linux-mobile 1
2011-08-30 13:32:37,327 INFO - Inserted the following build: mobile 7.0 linux-mobile [phone] Beta 1
2011-08-30 13:32:37,333 INFO - Did not find build entries in releases_raw table for mobile 7.0 20110816182137 Beta macosx-mobile 1
2011-08-30 13:32:37,334 INFO - Inserted the following build: mobile 7.0 macosx-mobile [phone] Beta 1
2011-08-30 13:32:37,341 INFO - Did not find build entries in releases_raw table for mobile 7.0 20110816165906 Beta win32-mobile 1
2011-08-30 13:32:37,343 INFO - Inserted the following build: mobile 7.0 win32-mobile [phone] Beta 1
2011-08-30 13:32:37,459 INFO - Did not find build entries in releases_raw table for mobile 7.0 20110824181241 Beta linux-android 2
2011-08-30 13:32:37,460 INFO - Inserted the following build: mobile 7.0 linux-android 20110824181241 Beta 2
2011-08-30 13:32:37,467 INFO - Did not find build entries in releases_raw table for mobile 7.0 20110824180046 Beta linux-maemo5-gtk 2
2011-08-30 13:32:37,469 INFO - Inserted the following build: mobile 7.0 linux-maemo5-gtk 20110824180046 Beta 2
2011-08-30 13:32:37,475 INFO - Did not find build entries in releases_raw table for mobile 7.0 20110824175006 Beta linux-mobile 2
2011-08-30 13:32:37,477 INFO - Inserted the following build: mobile 7.0 linux-mobile [phone] Beta 2
2011-08-30 13:32:37,485 INFO - Did not find build entries in releases_raw table for mobile 7.0 20110824191413 Beta macosx-mobile 2
2011-08-30 13:32:37,487 INFO - Inserted the following build: mobile 7.0 macosx-mobile [phone] Beta 2
2011-08-30 13:32:37,497 INFO - Did not find build entries in releases_raw table for mobile 7.0 20110824183137 Beta win32-mobile 2
2011-08-30 13:32:37,500 INFO - Inserted the following build: mobile 7.0 win32-mobile [phone] Beta 2
2011-08-30 13:32:37,532 INFO - Did not find build entries in releases_raw table for mobile 7.0 20110830105547 Beta linux-android 3
2011-08-30 13:32:37,534 INFO - Inserted the following build: mobile 7.0 linux-android 20110830105547 Beta 3
2011-08-30 13:32:37,539 INFO - Did not find build entries in releases_raw table for mobile 7.0 20110830103857 Beta linux-maemo5-gtk 3
2011-08-30 13:32:37,541 INFO - Inserted the following build: mobile 7.0 linux-maemo5-gtk 20110830103857 Beta 3
2011-08-30 13:32:37,548 INFO - Did not find build entries in releases_raw table for mobile 7.0 20110830111038 Beta linux-mobile 3
2011-08-30 13:32:37,550 INFO - Inserted the following build: mobile 7.0 linux-mobile [phone] Beta 3
2011-08-30 13:32:37,556 INFO - Did not find build entries in releases_raw table for mobile 7.0 20110830115410 Beta macosx-mobile 3
2011-08-30 13:32:37,558 INFO - Inserted the following build: mobile 7.0 macosx-mobile [phone] Beta 3
2011-08-30 13:32:37,564 INFO - Did not find build entries in releases_raw table for mobile 7.0 20110830115713 Beta win32-mobile 3
2011-08-30 13:32:37,566 INFO - Inserted the following build: mobile 7.0 win32-mobile [phone] Beta 3

This makes me wonder if we shouldn't even bother matching the platform names against a list, and just assume whatever comes before "_info.txt" is a platform name. I left this in place since that's how the nightly version works.

Comment 6

[Lonnen :lonnen]

Comment on attachment 556975 [details] [diff] [review]
add new mobile platform names

looks good

Comment 7

[Robert Helmer [:rhelmer]]

Trunk:
Committed revision 3488.

Branch:
Committed revision 3487.

Thanks lonenn!

Comment 8

[Robert Helmer [:rhelmer]]

For QA benefit - I think comment 5 serves as verification (I ran it by hand, if we're shipping 2.2.2 today we don't have time to wait for stage cron to run unfortunately).

We already verified the cron changes on stage though so I am pretty sure that will still work :)

Comment 9

[Matt Brandt [:mbrandt]]

thx rhelmer (qa <3's benifits). Bumping to QA verified per comment 5.

Comment 10

[[e-mail] (:bhearsum)]

Just to be clear: RelEng doesn't need to e-mail socorro-dev for releases anymore, right?

Comment 11

[Robert Helmer [:rhelmer]]

(In reply to Ben Hearsum [:bhearsum] from comment #10)
> Just to be clear: RelEng doesn't need to e-mail socorro-dev for releases
> anymore, right?

Nope, we're all set now thanks!