986487 - release repacks needs to submit data to balrog

Status: RESOLVED FIXED

(Release Engineering :: Release Automation: Other, defect)

Description

[[e-mail] (:bhearsum)]

We got this for free for en-US release builds, because they share a Factory with nightlies. Unfortunately, release repacks happen through entirely different means than nightly repacks, so we'll need to implement this separately.

Hopefully we can just import something (SingleLocale?) from lib/python/balrog/submitter and call it with the right args.

Comment 1

[[e-mail] (:bhearsum)]

Attachment: wip, partly tested

I tested this patch by hand on a linux build machine and it worked :). I haven't tested on other platforms, or inside of Buildbot. I'd like to get feedback on the approach before I do further testing.

In general, I think this patch is hacky. I'm not a big fan of returning the checksums file from repackLocale, but the only other idea I could come up with doing all the balrog stuff inside of it, which I think is even worse.

This patch also gives us a hard dependency on buildid, which comes in via a scheduler. This means that we need buildid to be set when using standalone repack builders. With already set 3 properties for them when we trigger, so unless we find a way to get rid of one, we can't also set buildid. They're important builders, so I don't think this can be landed without figuring that out. I'm hoping that I can either stop setting script_repo_revision or release_tag - I'm not 100% sure they're both necessary anyways.

Comment 2

[[e-mail] (:bhearsum)]

Attachment: release.py changes, untested

Comment 3

[[e-mail] (:bhearsum)]

Oh, here's what was submitted in my test:
https://aus4-admin-dev.allizom.org/releases/Firefox-29.0b4-build1/data

And I ran it like this:
bash-4.1$ python scripts/scripts/l10n/create-release-repacks.py -c mozilla/production_config.py -r mozilla/release-firefox-mozilla-beta.py -b http://hg.mozilla.org/build/buildbot-configs -t FIREFOX_29_0b4_RELEASE -p linux64 --locale de --stage-ssh-key ffxbld_dsa --stage-server dev-stage01.srv.releng.scl3.mozilla.com --stage-username ffxbld --hghost hg.mozilla.org --compare-locales-repo-path build/compare-locales --generate-partials --tooltool-manifest browser/config/tooltool-manifests/linux64/releng.manifest --tooltool-script /builds/tooltool.py --tooltool-url http://runtime-binaries.pvt.build.mozilla.org/tooltool --balrog-api-root https://aus4-admin-dev.allizom.org --credentials-file oauth.txt --balrog-username stage-ffxbld --buildid 20140101010101

Comment 4

[Rail Aliiev [:rail]]

Comment on attachment 8400130 [details] [diff] [review]
wip, partly tested

Review of attachment 8400130 [details] [diff] [review]:
-----------------------------------------------------------------

::: scripts/l10n/create-release-repacks.py
@@ +135,5 @@
> +                    raise Exception("Couldn't find complete mar info")
> +                balrog_submitter.run(platform, product.capitalize(), appVersion, version,
> +                                    buildNumber, l, balrog_hash, appVersion,
> +                                    buildid, marInfo['complete']['hash'],
> +                                    marInfo['complete']['size'])

Too many arguments, easy to mess. Can you pass them named?

@@ -217,5 @@
>          platform = "linux32"
>      mozconfig = path.join(sourceRepoInfo['name'], releaseConfig["appName"],
>                            "config", "mozconfigs", platform,
>                            "l10n-mozconfig")
> -    # FIXME: please kill the following block when ESR17 is dead

Yay!

@@ +289,5 @@
>  
> +    if options.balrog_api_root:
> +        credentials = {}
> +        execfile(options.credentials_file, credentials)
> +        auth = (options.balrog_username, credentials['balrog_credentials'][options.balrog_username])

You can avoid execfile like this: http://hg.mozilla.org/build/tools/file/cc45ac0ffe01/scripts/release/stage-tasks.py#l385

Comment 5

[[e-mail] (:bhearsum)]

(In reply to Rail Aliiev [:rail] from comment #4)
> Comment on attachment 8400130 [details] [diff] [review]
> wip, partly tested
> 
> Review of attachment 8400130 [details] [diff] [review]:
> -----------------------------------------------------------------
> 
> ::: scripts/l10n/create-release-repacks.py
> @@ +135,5 @@
> > +                    raise Exception("Couldn't find complete mar info")
> > +                balrog_submitter.run(platform, product.capitalize(), appVersion, version,
> > +                                    buildNumber, l, balrog_hash, appVersion,
> > +                                    buildid, marInfo['complete']['hash'],
> > +                                    marInfo['complete']['size'])
> 
> Too many arguments, easy to mess. Can you pass them named?


Good call...this caught a bug where I passed hash size/value wrong already!

Comment 6

[[e-mail] (:bhearsum)]

Attachment: buildbotcustom changes, tested

New in this version:
* Merge with 990573 (copy_properties)
* Make standalone repacks possible by hardcoding release_config at config time. This requires a doc update after I land.

Comment 7

[[e-mail] (:bhearsum)]

Attachment: tested version of tools patch

I tested this with Firefox and Thunderbird repacks in dev. I wasn't able to test Thunderbird Windows repacks due to bug 909952, but the others all worked. Logs are available here:
http://dev-master1.srv.releng.scl3.mozilla.com:8018/builders/release-comm-beta-macosx64_repack_1%2F10/builds/3
http://dev-master1.srv.releng.scl3.mozilla.com:8018/builders/release-comm-beta-linux64_repack_1%2F10/builds/1
http://dev-master1.srv.releng.scl3.mozilla.com:8018/builders/release-mozilla-beta-win32_repack_1%2F10/builds/3
http://dev-master1.srv.releng.scl3.mozilla.com:8018/builders/release-mozilla-beta-win32_standalone_repack/builds/1
http://dev-master1.srv.releng.scl3.mozilla.com:8018/builders/release-mozilla-beta-linux64_standalone_repack/builds/3
http://dev-master1.srv.releng.scl3.mozilla.com:8018/builders/release-mozilla-beta-linux64_repack_1%2F10/builds/5
http://dev-master1.srv.releng.scl3.mozilla.com:8018/builders/release-mozilla-beta-macosx64_repack_1%2F10/builds/0

And Balrog blobs are here:
https://aus4-admin-dev.allizom.org/releases/Firefox-29.0b4-build1/data
https://aus4-admin-dev.allizom.org/releases/Thunderbird-28.0b1-build1/data

Changes since the last patch:
* Address review comments
* Fix checksums file parsing on Windows (need --no-print-directory there, because pymake prints by default, I guess)
* Don't purge oauth.txt!

Comment 8

[Rail Aliiev [:rail]]

Comment on attachment 8401265 [details] [diff] [review]
buildbotcustom changes, tested

Review of attachment 8401265 [details] [diff] [review]:
-----------------------------------------------------------------

::: process/release.py
@@ +72,5 @@
>          'base_clobber_url', branchConfig['base_clobber_url'])
> +    balrog_api_root=releaseConfig.get('balrog_api_root',
> +        branchConfig.get('balrog_api_root', None))
> +    balrog_username=releaseConfig.get('balrog_username',
> +        branchConfig.get('balrog_username', None))

A nit. Can you add spaces around = ?

Comment 9

[Rail Aliiev [:rail]]

Comment on attachment 8401270 [details] [diff] [review]
tested version of tools patch

Review of attachment 8401270 [details] [diff] [review]:
-----------------------------------------------------------------

sheeep it!

Comment 10

[[e-mail] (:bhearsum)]

Comment on attachment 8401265 [details] [diff] [review]
buildbotcustom changes, tested

Landed. Reconfig coming soon, in time for 29.0b6.

Comment 11

[[e-mail] (:bhearsum)]

in production

Comment 12

[Nick Thomas [:nthomas] (UTC+12)]

We'll need to add a retry around the submissions, particularly for locales. We hit this error about 15 times in 29.0b6, sometimes one a repack job, sometimes several:

Starting new HTTPS connection (1): aus4-admin.mozilla.org
Starting new HTTPS connection (2): aus4-admin.mozilla.org
Caught HTTPError: Couldn't update release: Failed to update row, old_data_version doesn't match current data_version

http://ftp.mozilla.org/pub/mozilla.org/firefox/candidates/29.0b6-candidates/build1/logs/release-mozilla-beta-linux64_repack_9-bm94-build1-build1.txt.gz

Comment 13

[[e-mail] (:bhearsum)]

(In reply to Nick Thomas [:nthomas] from comment #12)
> We'll need to add a retry around the submissions, particularly for locales.
> We hit this error about 15 times in 29.0b6, sometimes one a repack job,
> sometimes several:
> 
> Starting new HTTPS connection (1): aus4-admin.mozilla.org
> Starting new HTTPS connection (2): aus4-admin.mozilla.org
> Caught HTTPError: Couldn't update release: Failed to update row,
> old_data_version doesn't match current data_version
> 
> http://ftp.mozilla.org/pub/mozilla.org/firefox/candidates/29.0b6-candidates/
> build1/logs/release-mozilla-beta-linux64_repack_9-bm94-build1-build1.txt.gz

Yikes, sorry about that! This hasn't been an issue for nightlies, so I guess I assumed retries were baked in at a deep level. Looks like those are handled by the submitter being in a RetryingShellCommand though (https://github.com/mozilla/build-buildbotcustom/blob/master/process/factory.py#L1850). Should be an easy fix here.

Comment 14

[[e-mail] (:bhearsum)]

Attachment: retry balrog submission

I gave this a quick test by hand, seems to work:
bash-4.1$ PATH="/tools/buildbot/bin:/usr/local/bin:/usr/lib64/ccache:/bin:/usr/bin:/usr/local/sbin:/usr/sbin:/sbin:/tools/git/bin:/tools/python27/bin:/tools/python27-mercurial/bin:/home/cltbld/bin" python scripts/scripts/l10n/create-release-repacks.py -c mozilla/production_config.py -r mozilla/release-firefox-mozilla-beta.py -b http://hg.mozilla.org/users/bhearsum_mozilla.com/buildbot-configs -t FIREFOX_29_0b4_RELEASE -p linux64 --locale de --stage-ssh-key ffxbld_dsa --stage-server dev-stage01.srv.releng.scl3.mozilla.com --stage-username ffxbld --hghost hg.mozilla.org --compare-locales-repo-path build/compare-locales --generate-partials --tooltool-manifest browser/config/tooltool-manifests/linux64/releng.manifest --tooltool-script /builds/tooltool.py --tooltool-url http://runtime-binaries.pvt.build.mozilla.org/tooltool --balrog-api-root https://aus4-admin-dev.allizom.org --balrog-username=stage-ffxbld --credentials-file BuildSlaves.py --buildid=20140101010101

<snip>

retry: Calling <bound method ReleaseSubmitter.run of <balrog.submitter.cli.ReleaseSubmitter object at 0x170cb10>> with args: (), kwargs: {'completeMarSize': 35341442, 'buildID': '20140101010101', 'completeMarHash': 'a180d02202f986165c68dfe6ee50839302fb1994b17bdad65e4bd97da6c2935f8230d5465b4ffe58ecba4b530691d1830d31cf71cf46c7a962c419e4618a20dc', 'productName': 'Firefox', 'hashFunction': 'sha512', 'locale': 'de', 'build_number': 1, 'extVersion': '29.0', 'appVersion': '29.0', 'platform': 'linux64', 'version': '29.0b4'}, attempt #1
Starting new HTTPS connection (1): aus4-admin-dev.allizom.org
Starting new HTTPS connection (2): aus4-admin-dev.allizom.org

Comment 15

[[e-mail] (:bhearsum)]

No issues with repacks in 29.0b7. I think we're done here.

Comment 16

[Rail Aliiev [:rail]]

Looks like this broke ESR.

Comment 17

[Rail Aliiev [:rail]]

The following tracebacks were detected during repacks:
el:
Traceback (most recent call last):
  File "/builds/slave/rel-m-esr24-osx64_rpk_3-000000/scripts/scripts/l10n/create-release-repacks.py", line 128, in createRepacks
    checksums = parseChecksumsFile(open(checksums_file).read())
IOError: [Errno 2] No such file or directory: 'dist/mac/el/firefox-24.5.0esr.checksums'

Comment 18

[Rail Aliiev [:rail]]

Hmm, absObjdir at http://hg.mozilla.org/build/tools/rev/349972686306#l2.53 should be absolute, but it doesn't look like we make it absolute anywhere.

Comment 19

[Rail Aliiev [:rail]]

Reading the code tells me that we should upload everything we needed, we failed to submit updates to Balrog only. I'm going to proceed and mark all all l10n repacks ready to unblock the rest.

Comment 20

[[e-mail] (:bhearsum)]

(In reply to Rail Aliiev [:rail] from comment #18)
> Hmm, absObjdir at http://hg.mozilla.org/build/tools/rev/349972686306#l2.53
> should be absolute, but it doesn't look like we make it absolute anywhere.

Good catch, I think you're right...the variable name certainly says that it's _supposed_ to be abs, as well as the fact that normpath is used in repackLocale. I'll test a fix for this soon.

Comment 21

[[e-mail] (:bhearsum)]

Fixing the ESR bustage is high on my priority list, but I got pulled into even more important b2g work. This will definitely be fixed before the next scheduled release, but if have a chemspill or respin in the near future, this may still be an issue for it.

Comment 22

[[e-mail] (:bhearsum)]

Attachment: balrog-esr-repack-tools-3.diff

So, this problem was even weirder than I thought. I found a few issues:
1) I was joining absObjdir and relative_checksums. This means that we were coming up with paths like /builds/slave/dir/dist/... - I have no idea how this worked for non-ESR, except maybe related to the second point:
2) There was a leading single quote in the echoed checksums file.

The fix is a bit weird too, here's a summary:
* Actually make absObjdir an abspath.
* Adjust how absSourceRepoPath is calculated, because the existing way doesn't work with abspaths on Windows.
* Calculate checksums path based on localeSrcDir (the same directory we run make echo-variable from) to get the correct path to it.
* Remove dead code (localeSrcDir being passed to l10nRepackPrep)
* Print exceptions per locale, in addition to the summary at the end. Makes debugging easier/faster.3

Here's blobs from the successful submissions:
https://aus4-admin-dev.allizom.org/releases/Firefox-30.0b3-build1/data
https://aus4-admin-dev.allizom.org/releases/Thunderbird-30.0b1-build2/data
https://aus4-admin-dev.allizom.org/releases/Firefox-24.5.0esr-build1/data
https://aus4-admin-dev.allizom.org/releases/Thunderbird-24.5.0-build1/data

If you'd like to see the output from some runs, I can link you to my dev-master, too.

Comment 23

[[e-mail] (:bhearsum)]

OK, this should be fixed for good now.