562114 - Make Socorro's nightly buildID/changeset file scraper handle both current and new file format

Status: RESOLVED FIXED

(Socorro :: General, task)

Description

[Smokey Ardisson (offline for a while; not following bugs - do not email)]

Ted tells me that Socorro is now scraping the nightly builds' buildID+changeset file (as part of the changes for bug 549999, I presume).

We need to change that file so that it is useful to applications other than Firefox (i.e., every other Mozilla app, all of which are built from multiple repos), which unfortunately means changing the file format.

In order for this format change not to break Socorro, Socorro needs to first change its scraper to support the new file format *in addition* to the current one, and deploy that change to production.  After that, we'll start landing the new format across the several branches.

While the old format is

$BUILDID $CHANGESET

e.g.

20100427030621 29a6a85fab8e


The new format is going to be

$BUILDID
$PLATFORM_CHANGESET_URL
$APP_CHANGESET1_URL
$APP_CHANGESET2_URL
…

(where $APP_CHANGESET1_URL et seq. are "optional" and will only be used by Fennec, Thunderbird, SeaMonkey, and Camino)
e.g.

20100427030621
http://hg.mozilla.org/mozilla-central/rev/29a6a85fab8e

for a Firefox build and (eventually)

20100427004358
http://hg.mozilla.org/mozilla-central/rev/0b0b07662a88
http://hg.mozilla.org/comm-central/rev/8c4471d50289
http://hg.mozilla.org/chatzilla/rev/…
http://hg.mozilla.org/dom-inspector/rev/…
…

for a SeaMonkey build.

Comment 1

[Smokey Ardisson (offline for a while; not following bugs - do not email)]

Soccoro folks: any idea on when this might get scheduled for implementation?

Getting this implemented and deployed blocks my patch in bug 549958, which in turn blocks getting useful (and correct) changeset info stored with nightlies for regression-hunting for all our non-Firefox products.

Comment 2

[Smokey Ardisson (offline for a while; not following bugs - do not email)]

Austin/Laura/Ryan: ping again?  Can this make 1.8, which I understand is the next open release?

Comment 3

[Laura Thomson :laura]

Smokey, we're trying to reduce scope for 1.8 at the moment :( but it may be possible.  (1.7 is blocked on 3.6.4, and we're trying to ship 1.8 this quarter)

I'm not 100% clear on what needs doing for this bug - Ryan did bug 549999 so perhaps he can comment as to how much work is involved.  If he thinks he can get it done for 1.8, that's fine by me.  (The 1.8 critical path is back end.)

Comment 4

[Ryan Snyder [:ryansnyder] [:rsnyder] [:rysny]]

The nightly builds are currently being scraped for Firefox only at the moment, and include:
http://ftp.mozilla.org/pub/mozilla.org/firefox/nightly/latest-mozilla-1.9.1/
http://ftp.mozilla.org/pub/mozilla.org/firefox/nightly/latest-mozilla-1.9.2/
http://ftp.mozilla.org/pub/mozilla.org/firefox/nightly/latest-mozilla-central/

After looking at the code, if you make changes on your end it shouldn't affect the current build scraping script.  But the script will need to be changed in order to incorporate other products and the changes you're making to the config files.

Comment 5

[Smokey Ardisson (offline for a while; not following bugs - do not email)]

(In reply to comment #4)
> After looking at the code, if you make changes on your end it shouldn't affect
> the current build scraping script.  But the script will need to be changed in
> order to incorporate other products and the changes you're making to the config
> files.

Ryan, I'm not sure I understand this comment ;)  This is what I get out of it:

1) It's OK for me to go ahead make the change mentioned in comment 0, which reformats the nightly builds' buildID+changeset file; the new format won't break Socorro's scraper script's ability to get the nightly build ID from Firefox builds. 
2) But Socorro's script will need changes to do nightly scraping for products other than Firefox (I understand this part).
3) But I'm also changing some sort of config files (I am?), and Socorro's script will need changes to handle that.

I understand part 2.  Part 1 and Part 3 seem like they're in contradiction, though, and I don't know what "config files" I'm changing….

Comment 6

[Ryan Snyder [:ryansnyder] [:rsnyder] [:rysny]]

Oops, sorry Smokey. :)  When I mentioned the config files you'd be changing, I meant the build txt files.  But just to clarify:

Yes, it's okay for you to make the changes you've proposed.  This will not affect the current builds script on Socorro.

I will need to update the Socorro script to incorporate other products besides Firefox, and to ensure that the Socorro script handles $APP_CHANGESET1_URL and $APP_CHANGESET2_URL properly.

Comment 7

[Smokey Ardisson (offline for a while; not following bugs - do not email)]

Thanks, Ryan.

Just to be completely sure, though: the Socorro scraper is not currently doing anything with the changeset part of the build txt files (or is capable of handling changes to it)?  I ask because that is also changing, from a changeset value (29a6a85fab8e) to a changeset URL (http://hg.mozilla.org/mozilla-central/rev/29a6a85fab8e), in addition to moving on to its own line in the file.

Comment 8

[Ryan Snyder [:ryansnyder] [:rsnyder] [:rysny]]

Actually Smokey, thanks for double-checking on that.  I misread the 2nd value as remaining a value, rather than changing to an URL.  You're right, I will need to update the build scraping script accordingly.  

I'll hop on this shortly.  Code freeze for 1.8 is rapidly approaching, but I'll do what I can to get this change in.

Thanks again for pointing that out...

Comment 9

[Smokey Ardisson (offline for a while; not following bugs - do not email)]

One more thing that occurred to me that it would be wise to reiterate/clarify: the scraper will need to support *both* the current format and the new format for a while, until I can get bug 549958 approved and landed on all the branches; the likelihood of it happening all at once is very slim. ;)

Comment 10

[Ryan Snyder [:ryansnyder] [:rsnyder] [:rysny]]

Thanks Smokey, I will make sure that it handles both the current format and the new format.

FYI - The .txt files that the script is currently parsing is available for Seamonkey and Camino nightly builds.  But the .txt files are not currently located in the folders for Thunderbird and Fennec nightly builds:

http://ftp.mozilla.org/pub/mozilla.org/thunderbird/nightly/latest-comm-1.9.1/
http://ftp.mozilla.org/pub/mozilla.org/thunderbird/nightly/latest-comm-1.9.2/
http://ftp.mozilla.org/pub/mozilla.org/thunderbird/nightly/latest-comm-central-trunk/

http://ftp.mozilla.org/pub/mozilla.org/mobile/nightly/latest-mobile-1.9.2/
http://ftp.mozilla.org/pub/mozilla.org/mobile/nightly/latest-mobile-trunk/

Comment 11

[Smokey Ardisson (offline for a while; not following bugs - do not email)]

(In reply to comment #10)
> FYI - The .txt files that the script is currently parsing is available for
> Seamonkey and Camino nightly builds.  But the .txt files are not currently
> located in the folders for Thunderbird and Fennec nightly builds:

Yeah, I don't know what's going on there; I poked around a little bit when I was working on bug 549958 and didn't make any headway.

The files are produced automatically by the build system, so my best SWAG is that the upload of the files is done in some "custom" buildbot step that's being run by Firefox and SeaMonkey but not by Thunderbird or Fennec.

Comment 12

[(not currently active) Ted Mielczarek]

Firefox (at least) uses "make upload" which is implemented in the build system:
http://mxr.mozilla.org/mozilla-central/source/toolkit/mozapps/installer/packager.mk#534
it knows to upload all the relevant bits.

Comment 13

[Mark Banner (:standard8)]

So if this is make upload, I suspect Thunderbird hasn't transitioned to it yet. We should be upgrading it to the latest stuff soon... but talk to gozer to find out more.

Comment 14

[Ryan Snyder [:ryansnyder] [:rsnyder] [:rysny]]

Attachment: Patch 1 for 562114

This patch enables build files to be scraped for Firefox and Seamonkey.  Changes are prepped for Thunderbird, Camino and Fennec, and the script will automatically start scraping these files when the appropriate changes have been made on the platforms' ends.  This patch will now scrape the platform url/changeset, and the 2 app changesets urls that will be added in the near future.

It requires the following changes to the builds table:
{{{
ALTER TABLE builds RENAME COLUMN changeset TO platform_changeset;
ALTER TABLE builds ADD app_changeset_1 text;
ALTER TABLE builds ADD app_changeset_2 text;
}}}

Commonconfig.py changes are detailed here in the Socorro Upgrade doc:
http://code.google.com/p/socorro/wiki/SocorroUpgrade#commonconfig.py

Comment 15

[Ryan Snyder [:ryansnyder] [:rsnyder] [:rysny]]

Attachment: PHP Patch for 562114

This patch is a small change the the builds links, which adds the product and version to each of the links.

Comment 16

[Austin King [:ozten]]

Comment on attachment 460674 [details]
PHP Patch for 562114

Looks good! r+. The SeaMonkey hardcoded case is wonky for non-Mozilla installs. They may have camel-case products also. Eventually we should pull this out and use a product config (from DB or config file).

Comment 17

[Ryan Snyder [:ryansnyder] [:rsnyder] [:rysny]]

Thanks Austin.  Thanks Lars.  Removed an unnecessary function per Lars' suggestion.  Haven't decided the best way to handle a product named any of 4 ways - seamonkey, Seamonkey, SeaMonkey, Sea Monkey - but it goes beyond just this page and is found throughout the database and site.

This will roll with 1.8, which I don't believe has an official release date just yet.

==

Sending        scripts/config/buildsconfig.py.dist
Sending        scripts/startBuilds.py
Sending        socorro/cron/builds.py
Sending        socorro/database/schema.py
Sending        socorro/unittest/cron/testBuilds.py
Sending        webapp-php/application/views/common/product_builds.php
Transmitting file data ......
Committed revision 2256.

Comment 18

[Smokey Ardisson (offline for a while; not following bugs - do not email)]

Thanks, Ryan et al.!

(In reply to comment #17)
> This will roll with 1.8, which I don't believe has an official release date
> just yet.

I don't see a tracking bug for deploying it yet, either.  I'll keep on the look-out for it.

Comment 19

[Ryan Snyder [:ryansnyder] [:rsnyder] [:rysny]]

It sounds like we're looking at ~2 weeks until 1.8 rolls out, but we're still trying to figure out the exact push date.

Comment 20

[Ryan Snyder [:ryansnyder] [:rsnyder] [:rysny]]

Smokey - Sorry for the late notice, but we're in the process of rolling out this afternoon.  We hadn't solidified a rollout date / time until lunchtime today.  

These scripts are set to pull current and new file formats, so it will be okay if your code doesn't ship immediately.  Thanks!

Comment 21

[Smokey Ardisson (offline for a while; not following bugs - do not email)]

Thanks, Ryan; I'll start requesting approvals :)

Comment 22

[Ryan Snyder [:ryansnyder] [:rsnyder] [:rysny]]

Smokey - We're rolling back because of issues that we're having with crash report submissions.  We are hoping to push this release out the door again either next week or the following, once we've got a handle on the issue and have it resolved in a staging environment.  Sorry for the inconvenience.

Comment 23

[Smokey Ardisson (offline for a while; not following bugs - do not email)]

Thanks for the update :(

Comment 24

[Laura Thomson :laura]

Merge to 1.7.6

Comment 25

[Ryan Snyder [:ryansnyder] [:rsnyder] [:rysny]]

Smokey - FYI, the goal date for getting this implemented in the next release is Dec 1.

Comment 26

[Smokey Ardisson (offline for a while; not following bugs - do not email)]

Thanks for the update, Ryan.  I have approval on the patch for bug 549958, so assuming the tree is in a state where non-blockers can land, I'll look forward to landing it with 1.7.6 then. :)

Comment 27

[Ryan Snyder [:ryansnyder] [:rsnyder] [:rysny]]

Updating the code according to 1.7.6 changes on the middleware layer.

This will need to be pushed to stage for testing.  Once subversion has been updated on stage, we will need to run scripts/startBuilds.py.

Documentation has been added to https://code.google.com/p/socorro/wiki/SocorroUpgrade

Smokey, this is slated to rollout to production Thursday, Dec 2.

== 

Sending        scripts/config/buildsconfig.py.dist
Sending        scripts/startBuilds.py
Sending        socorro/cron/builds.py
Sending        socorro/database/schema.py
Sending        socorro/unittest/cron/testBuilds.py
Sending        webapp-php/application/views/common/product_builds.php
Transmitting file data ......
Committed revision 2772.

Comment 28

[Ryan Snyder [:ryansnyder] [:rsnyder] [:rysny]]

Smokey, this is now currently slated to rollout to production on Thursday, Dec 9.

Comment 29

[Ryan Snyder [:ryansnyder] [:rsnyder] [:rysny]]

And ... this is now slated to rollout to production sometime in early 2011.

Comment 30

[Ryan Snyder [:ryansnyder] [:rsnyder] [:rysny]]

This is now in production, now that the data center has moved from SJC to PHX.

Comment 31

[Smokey Ardisson (offline for a while; not following bugs - do not email)]

Woot!  Thanks, Ryan!

Comment 32

[Smokey Ardisson (offline for a while; not following bugs - do not email)]

FYI, I landed bug 549958 last night on mozilla-central, so today's Firefox, Fennec, Thunderbird, and SeaMonkey nightlies will be using the new format.

(Branch builds will still be using the old format, though I hope to get approval to switch them in the release cycle after Gecko 1.9.2.14/1.9.1.16.)

Comment 33

[Ryan Snyder [:ryansnyder] [:rsnyder] [:rysny]]

Awesome, thanks Smokey.

Comment 34

[Smokey Ardisson (offline for a while; not following bugs - do not email)]

Looks like I forgot to mention it here, but bug 549958 landed on the 1.9.1 and 1.9.2 branches on March 1, so you can officially consider the old "buildID changeset" format obsolete :)