1338960 - Modify automation to use balrog and not aus2

Status: RESOLVED INVALID

(SeaMonkey :: Release Engineering, defect)

Description

[Edmund Wong (:ewong)]

We need to change the buildbotcustom code for the ReleaseUpdatesFactory to
use balrog and not upload stuff to aus2.

Comment 1

[Edmund Wong (:ewong)]

A quick note (for mostly to jog my memory should I forget what needs to be
done):

We currently upload snippets/partial-mars to aus2.

What we need to do now:

1) Upload the partialmars/snippets to the upload host (which the files need to be 
directed to the right release/beta candidates/ filepath) i.e. candidates/SeaMonkey-<ver>-build<build#>/..

2) Gather *all* the necessary release information from the properties gathered
from all the previous steps(build, repacks and snippet creations, etc) and
build a balrog dictionary. [this bug, bug 1338958, and 1338953 *should* help with 
this
  - basically build info (package url+size+hash), [bug 1338958]
  - locale repack info (locale package size + hash) [bug 1338953]
  - partial snippets (size + hash and from version) [this bug]

3) Push the balrog release information *to* the balrog server as the new
release version. [dep : bug 1338967]

4) Update the balrog release/beta rule to map to the new release. [dep. bug 1338967]

Comment 3

[Edmund Wong (:ewong)]

Attachment: [custom] for locale-repack

Comment 4

[Edmund Wong (:ewong)]

Comment on attachment 8836980 [details] [diff] [review]
[custom] for locale-repack

Comment from dup'd bug:

Balrog release information require locale complete.mar (and partial.mar) 
properties (checksums, filesize, and old version info amongst others).

most probably need to be before the make_upload step

Comment 7

[Edmund Wong (:ewong)]

For beta/release, make_upload sets the following properties:

symbolsUrl and packageUrl

However, packageUrl is set to the wrong file.

packageUrl: 'https://archive.mozilla.org/pub/seamonkey/candidates/2.46-candidates/build9/linux-i686/en-US/mozharness.zip'

This pretty much reflects the problem we have with our tests; but,
this needs to be fixed as well as adding other properties:

packageUrl, size, hash and whatever else is necessary for 
release and balrog.

Comment 8

[Edmund Wong (:ewong)]

Attachment: [tools] adding balrog scripts to tools repo.

Straight 'fork' of the moco version.

Comment 9

[Frank-Rainer Grahl (:frg)]

Comment on attachment 8836986 [details] [diff] [review]
[tools] adding balrog scripts to tools repo.

Don't have the means to test it so just did go over it for formalities.

> if options.schema_version == 3:
> +            submitter = NightlySubmitterV3(options.api_root, auth,
> +                                           options.dummy,
> +                                           url_replacements=url_replacements)

With a new file is version 3 still valid? 

> +FTP_SERVER_TEMPLATE = 'http://%s/pub/mozilla.org'

https?

Comment 10

[Edmund Wong (:ewong)]

(In reply to Frank-Rainer Grahl from comment #9)
> Comment on attachment 8836986 [details] [diff] [review]
> [tools] adding balrog scripts to tools repo.
> 
> Don't have the means to test it so just did go over it for formalities.
> 
> > if options.schema_version == 3:
> > +            submitter = NightlySubmitterV3(options.api_root, auth,
> > +                                           options.dummy,
> > +                                           url_replacements=url_replacements)
> 
> With a new file is version 3 still valid? 

Point taken.  Since any imports we do will most likely be of schema_version =4,
this part would be irrelevant.

> 
> > +FTP_SERVER_TEMPLATE = 'http://%s/pub/mozilla.org'
> 
> https?

Ah right.. and it's no longer with mozilla.org.

Comment 11

[Edmund Wong (:ewong)]

There is a slight problem with the scripts as they are based on automation.  They won't work on the loaner unfortunately, so I'll need to either modify the scripts or create a new one based on the fact the loaner cannot get (easily) information from the master.

I think my approach will be to create a new set of required scripts (as of this
writing, just balrog-submitter.py) to use different commandline arguments.

Comment 12

[Edmund Wong (:ewong)]

Attachment: [tools] adding balrog scripts to tools repo. (v2)

I'll most likely push the reviewed contents so that I can test it out in nightly.

Comment 13

[Frank-Rainer Grahl (:frg)]

Comment on attachment 8837943 [details] [diff] [review]
[tools] adding balrog scripts to tools repo. (v2)

review postponed as per irc

Comment 14

[Edmund Wong (:ewong)]

Attachment: [tools] add balrog scripts to tools repo (v3)

Mostly code from moco's tools repo, with an additional script to gather the stuff in the loaner's env to a buildprops_balrog.json file which is then used by balrog-*.  

NB: There is a chance that I might just roll something that's a lot simpler for us. (although we *will* need the lib/python changes)

Comment 15

[Frank-Rainer Grahl (:frg)]

Comment on attachment 8840336 [details] [diff] [review]
[tools] add balrog scripts to tools repo (v3)

As stated on irc just a formal check and even this one only lightly.

+    return 'https://www.mozilla.org/%%LOCALE%%/%s/%s/releasenotes/' % (product, appVersion)

releasenotes for SeaMonkey are not on mozilla.org. Is this one checked / read during building?

Comment 16

[Edmund Wong (:ewong)]

Attachment: [configs] update configs to allow balrog submission

the release steps need to include balrog information.

Comment 17

[Edmund Wong (:ewong)]

Attachment: [custom] add balrog submission steps

add submitBalrogUpdates step.  [current wip]

Comment 18

[Edmund Wong (:ewong)]

to note, the configs and custom code patches will be manually applied to
the master so i can get the 2.48b1 running.  So these are wip so naturally,
2.48b1 will have bustages along the way; but thanks to 2.46, doing more
builds won't freak me out. :)

Comment 19

[Edmund Wong (:ewong)]

Attachment: [tools] Balrog scripts (v4)

this new set of scripts will be used instead.

Comment 20

[Frank-Rainer Grahl (:frg)]

Comment on attachment 8842288 [details] [diff] [review]
[custom] add balrog submission steps

> self.env.get('PYTHON26', 'python'),

Is this PYTHON26 or PYTHON27 after the other updates?

Comment 21

[Frank-Rainer Grahl (:frg)]

Comment on attachment 8842290 [details] [diff] [review]
[tools] Balrog scripts (v4)

I have faith in you here :)

Comment 22

[Edmund Wong (:ewong)]

(In reply to Frank-Rainer Grahl from comment #20)
> Comment on attachment 8842288 [details] [diff] [review]
> [custom] add balrog submission steps
> 
> > self.env.get('PYTHON26', 'python'),
> 
> Is this PYTHON26 or PYTHON27 after the other updates?

I think it's high time I change that to PYTHON27..  but it's ok.

Comment 23

[Edmund Wong (:ewong)]

Attachment: [puppet] need to set up balrog.token in puppet. [plr]

yup.. completely missed this required patch :(

Comment 24

[Edmund Wong (:ewong)]

Attachment: [tools] need to add cert bundle to allow https usage. [plr]

Comment 25

[Edmund Wong (:ewong)]

NB:

[exp] = patches that I will ask for review once I know for a fact they are
correct.

Comment 26

[Edmund Wong (:ewong)]

Comment on attachment 8842736 [details] [diff] [review]
[tools] need to add cert bundle to allow https usage. [plr]

Ignore previous comment.  I'll just set the patches to post-land-review [plr]
so that I can get 2.48b1 done sooner.

Comment 27

[Edmund Wong (:ewong)]

Attachment: [tools] fix scripts to enable https requests, and clean up balrog scripts. [plr]

Comment 28

[Frank-Rainer Grahl (:frg)]

Comment on attachment 8842742 [details] [diff] [review]
[tools] fix scripts to enable https requests, and clean up balrog scripts.  [plr]

Shouldn't the pyc files be in the ignore list?

Comment 29

[Edmund Wong (:ewong)]

Attachment: [custom] add balrog submission steps (v2)

replace the use of python26 and use PythonWjson  (as this is usually
the one we need).

Comment 30

[Edmund Wong (:ewong)]

Pushed to buildbotcustom:
https://hg.mozilla.org/build/buildbotcustom/rev/bef988459374c33c6ee7b9765592613f31707a80

Comment 31

[Edmund Wong (:ewong)]

Attachment: [custom] fix for productName and buildNumber

Comment 32

[Edmund Wong (:ewong)]

Attachment: [tools] remove _save_json() usage [remenant from test phase]

Comment 33

[Edmund Wong (:ewong)]

Attachment: [tools] remove _save_json() usage [remnant from test phase]

Comment 34

[Edmund Wong (:ewong)]

Attachment: [custom] fix for using the wrong parameters and python.

was using the wrong parameter to specify the credentials and
missed the linux64 detection of the right python.

Comment 35

[Frank-Rainer Grahl (:frg)]

Comment on attachment 8849830 [details] [diff] [review]
[custom] fix for using the wrong parameters and python.

>>   elif platform in ('linux'):

Would it make sense to add windows64 too even if not used right now or is there no separate platform for it?

Comment 36

[Edmund Wong (:ewong)]

Attachment: [tools] use the properties' appVersion

Comment 37

[Edmund Wong (:ewong)]

Attachment: [tools] missing detection for - in platforms

Comment 38

[Edmund Wong (:ewong)]

Attachment: [configs] use stage-seabld for now

previously was seabld..  balrog stage uses stage-seabld.

Comment 39

[Edmund Wong (:ewong)]

I'm changing tact.  I can't test out balrog if we do it only for release so I"m going to 
get our nightlies to use balrog first (well, I'm going to do that with the regular builds
first.. they get built far more frequently so I can test it out more frequently)

while all the work I've done so far was geared towards release builds,  it shouldn't matter.
I just need to 'clean' up the current config, custom and tools code.

Comment 40

[Edmund Wong (:ewong)]

Attachment: [custom] buildbotcustom code fix to support nightly

Comment 41

[Edmund Wong (:ewong)]

Attachment: [tools] fix scripts to submit nightly info.

Comment 42

[Edmund Wong (:ewong)]

Attachment: [configs] add balrog_* fields to configs.

Comment 43

[Edmund Wong (:ewong)]

Attachment: [configs] add balrog_* fields to configs. to support both nightly and release

Comment 44

[Edmund Wong (:ewong)]

Attachment: [custom] buildbotcustom code fix to support nightly and release

Comment 45

[Edmund Wong (:ewong)]

Attachment: [tools] fix scripts to submit nightly and release info [wip]

seems to work with nightly, but I have no release data to test with.
Will try with 2.49.1

Comment 46

[Edmund Wong (:ewong)]

A few items of interest:

While the scripts 'work' with osx64 and linux*, it's not going to work well with
Win32 due to it being a manual process and having no access to the master and
thusly cannot get the properties.

I'll need to manually generate the properties file to pass to the balrog-submit.py
script.

Comment 47

[Edmund Wong (:ewong)]

Attachment: [custom] buildbotcustom code fix to support nightly and release

this patch also includes a snippet which disables clobbering of
the build directory after everything has been done.

Comment 48

[Edmund Wong (:ewong)]

Attachment: [custom] buildbotcustom code fix to support nightly, regular, debug and release [checked-in]

Comment 49

[Edmund Wong (:ewong)]

Attachment: [configs] add balrog_* fields to configs to support nightly,regular,debug and release [checked-in]

Comment 50

[Edmund Wong (:ewong)]

Attachment: [tools] fix scripts to submit nightly, regular, debug and release info [wip] [checked-in]

Comment 51

[Frank-Rainer Grahl (:frg)]

I only gave the three a lookover. Hard to review them properly without being able to try them. Thanks and hoping for the best.

Comment 52

[Edmund Wong (:ewong)]

Pushed also two bustage fixes:

https://hg.mozilla.org/build/buildbotcustom/rev/c29cd53299f5

and:

https://hg.mozilla.org/build/buildbot-configs/rev/994aab56e622

Comment 53

[Edmund Wong (:ewong)]

Attachment: [custom] Enable balrog submissions only for Linux32 nightly for now [checked-in]

Comment 54

[Edmund Wong (:ewong)]

Attachment: [configs] enable Linux32 nightly balrog submissions only. [checked-in]

Comment 55

[Frank-Rainer Grahl (:frg)]

Comment on attachment 8932037 [details] [diff] [review]
[configs] enable Linux32 nightly balrog submissions only. [checked-in]

>   'tooltool_token': 'e:/builds/tooltool.token',
> + 'balrog_credentials_file': 'c:/builds/balrog.token',
>   'balrog_submitter_extra_args': [],

c:\builds and not e:\builds is correct for Windows? Otherwise r+

Comment 56

[Frank-Rainer Grahl (:frg)]

Comment on attachment 8932035 [details] [diff] [review]
[custom] Enable balrog submissions only for Linux32 nightly for now [checked-in]

r+

Comment 57

[Edmund Wong (:ewong)]

(In reply to Frank-Rainer Grahl (:frg) from comment #55)
> Comment on attachment 8932037 [details] [diff] [review]
> [configs] enable Linux32 nightly balrog submissions only.
> 
> >   'tooltool_token': 'e:/builds/tooltool.token',
> > + 'balrog_credentials_file': 'c:/builds/balrog.token',
> >   'balrog_submitter_extra_args': [],
> 
> c:\builds and not e:\builds is correct for Windows? Otherwise r+

E: is for our current Win32 builders (the out-dated systems).
C: is for the loaner and possibly the future win* builders.  Our
win2003 builders are E:\builds based.

So yeah, some confusion there.  which is why the mozconfigs were
pointing to E:/ awhile back.

Comment 58

[Edmund Wong (:ewong)]

Comment on attachment 8842721 [details] [diff] [review]
[puppet] need to set up balrog.token in puppet. [plr]

ends up this isn't even needed and I had already pushed the patch.

Basically we download the file from the master..  so we don't need
to keep that many copies of this.

Comment 59

[Edmund Wong (:ewong)]

Attachment: [tools] clean up balrog submission scripts.

Comment 60

[Edmund Wong (:ewong)]

Attachment: [configs] enable nightly balrog submissions for the rest of the platforms [checked-in]

Comment 61

[Edmund Wong (:ewong)]

Attachment: [tools] clean up balrog submission scripts. [fix gen props for Win32] [checked-in]

don't remember if I mentioned this.. the gen-props script is to 
generate the properties json file for the loaner and won't be used
with whatever future Win* platforms.  It's just that the loaner
has no access to the master and thusly can't grab the properties.

Comment 62

[Edmund Wong (:ewong)]

Comment on attachment 8933529 [details] [diff] [review]
[configs] enable nightly balrog submissions for the rest of the platforms [checked-in]

Pushed to buildbot-configs:
https://hg.mozilla.org/build/buildbot-configs/rev/b9476e37f094

Comment 63

[Edmund Wong (:ewong)]

Comment on attachment 8933550 [details] [diff] [review]
[tools] clean up balrog submission scripts. [fix gen props for Win32] [checked-in]

Pushed to tools:
https://hg.mozilla.org/users/Callek_gmail.com/tools/rev/95fd9817778e

Comment 64

[Edmund Wong (:ewong)]

Attachment: [tools] fix locale update overwritting en_US locale

Comment 65

[Edmund Wong (:ewong)]

Attachment: [custom] move balrog submissions to after snippet generations

Comment 66

[Frank-Rainer Grahl (:frg)]

Comment on attachment 8935371 [details] [diff] [review]
[custom] move balrog submissions to after snippet generations

thanks

Comment 67

[Edmund Wong (:ewong)]

SeaMonkey won't be using Balrog, so closing.