Note: There are a few cases of duplicates in user autocompletion which are being worked on.

release repacks needs to submit data to balrog

RESOLVED FIXED

Status

Release Engineering
Release Automation
RESOLVED FIXED
3 years ago
3 years ago

People

(Reporter: bhearsum, Assigned: bhearsum)

Tracking

Firefox Tracking Flags

(Not tracked)

Details

Attachments

(4 attachments, 2 obsolete attachments)

(Assignee)

Description

3 years ago
We got this for free for en-US release builds, because they share a Factory with nightlies. Unfortunately, release repacks happen through entirely different means than nightly repacks, so we'll need to implement this separately.

Hopefully we can just import something (SingleLocale?) from lib/python/balrog/submitter and call it with the right args.
(Assignee)

Comment 1

3 years ago
Created attachment 8400130 [details] [diff] [review]
wip, partly tested

I tested this patch by hand on a linux build machine and it worked :). I haven't tested on other platforms, or inside of Buildbot. I'd like to get feedback on the approach before I do further testing.

In general, I think this patch is hacky. I'm not a big fan of returning the checksums file from repackLocale, but the only other idea I could come up with doing all the balrog stuff inside of it, which I think is even worse.

This patch also gives us a hard dependency on buildid, which comes in via a scheduler. This means that we need buildid to be set when using standalone repack builders. With already set 3 properties for them when we trigger, so unless we find a way to get rid of one, we can't also set buildid. They're important builders, so I don't think this can be landed without figuring that out. I'm hoping that I can either stop setting script_repo_revision or release_tag - I'm not 100% sure they're both necessary anyways.
Assignee: nobody → bhearsum
Attachment #8400130 - Flags: feedback?(rail)
(Assignee)

Comment 2

3 years ago
Created attachment 8400131 [details] [diff] [review]
release.py changes, untested
Attachment #8400131 - Flags: feedback?(rail)
(Assignee)

Comment 3

3 years ago
Oh, here's what was submitted in my test:
https://aus4-admin-dev.allizom.org/releases/Firefox-29.0b4-build1/data

And I ran it like this:
bash-4.1$ python scripts/scripts/l10n/create-release-repacks.py -c mozilla/production_config.py -r mozilla/release-firefox-mozilla-beta.py -b http://hg.mozilla.org/build/buildbot-configs -t FIREFOX_29_0b4_RELEASE -p linux64 --locale de --stage-ssh-key ffxbld_dsa --stage-server dev-stage01.srv.releng.scl3.mozilla.com --stage-username ffxbld --hghost hg.mozilla.org --compare-locales-repo-path build/compare-locales --generate-partials --tooltool-manifest browser/config/tooltool-manifests/linux64/releng.manifest --tooltool-script /builds/tooltool.py --tooltool-url http://runtime-binaries.pvt.build.mozilla.org/tooltool --balrog-api-root https://aus4-admin-dev.allizom.org --credentials-file oauth.txt --balrog-username stage-ffxbld --buildid 20140101010101
Comment on attachment 8400130 [details] [diff] [review]
wip, partly tested

Review of attachment 8400130 [details] [diff] [review]:
-----------------------------------------------------------------

::: scripts/l10n/create-release-repacks.py
@@ +135,5 @@
> +                    raise Exception("Couldn't find complete mar info")
> +                balrog_submitter.run(platform, product.capitalize(), appVersion, version,
> +                                    buildNumber, l, balrog_hash, appVersion,
> +                                    buildid, marInfo['complete']['hash'],
> +                                    marInfo['complete']['size'])

Too many arguments, easy to mess. Can you pass them named?

@@ -217,5 @@
>          platform = "linux32"
>      mozconfig = path.join(sourceRepoInfo['name'], releaseConfig["appName"],
>                            "config", "mozconfigs", platform,
>                            "l10n-mozconfig")
> -    # FIXME: please kill the following block when ESR17 is dead

Yay!

@@ +289,5 @@
>  
> +    if options.balrog_api_root:
> +        credentials = {}
> +        execfile(options.credentials_file, credentials)
> +        auth = (options.balrog_username, credentials['balrog_credentials'][options.balrog_username])

You can avoid execfile like this: http://hg.mozilla.org/build/tools/file/cc45ac0ffe01/scripts/release/stage-tasks.py#l385
Attachment #8400130 - Flags: feedback?(rail) → feedback+
Attachment #8400131 - Flags: feedback?(rail) → feedback+
(Assignee)

Comment 5

3 years ago
(In reply to Rail Aliiev [:rail] from comment #4)
> Comment on attachment 8400130 [details] [diff] [review]
> wip, partly tested
> 
> Review of attachment 8400130 [details] [diff] [review]:
> -----------------------------------------------------------------
> 
> ::: scripts/l10n/create-release-repacks.py
> @@ +135,5 @@
> > +                    raise Exception("Couldn't find complete mar info")
> > +                balrog_submitter.run(platform, product.capitalize(), appVersion, version,
> > +                                    buildNumber, l, balrog_hash, appVersion,
> > +                                    buildid, marInfo['complete']['hash'],
> > +                                    marInfo['complete']['size'])
> 
> Too many arguments, easy to mess. Can you pass them named?


Good call...this caught a bug where I passed hash size/value wrong already!
(Assignee)

Comment 6

3 years ago
Created attachment 8401265 [details] [diff] [review]
buildbotcustom changes, tested

New in this version:
* Merge with 990573 (copy_properties)
* Make standalone repacks possible by hardcoding release_config at config time. This requires a doc update after I land.
Attachment #8400130 - Attachment is obsolete: true
Attachment #8400131 - Attachment is obsolete: true
Attachment #8401265 - Flags: review?(rail)
(Assignee)

Comment 7

3 years ago
Created attachment 8401270 [details] [diff] [review]
tested version of tools patch

I tested this with Firefox and Thunderbird repacks in dev. I wasn't able to test Thunderbird Windows repacks due to bug 909952, but the others all worked. Logs are available here:
http://dev-master1.srv.releng.scl3.mozilla.com:8018/builders/release-comm-beta-macosx64_repack_1%2F10/builds/3
http://dev-master1.srv.releng.scl3.mozilla.com:8018/builders/release-comm-beta-linux64_repack_1%2F10/builds/1
http://dev-master1.srv.releng.scl3.mozilla.com:8018/builders/release-mozilla-beta-win32_repack_1%2F10/builds/3
http://dev-master1.srv.releng.scl3.mozilla.com:8018/builders/release-mozilla-beta-win32_standalone_repack/builds/1
http://dev-master1.srv.releng.scl3.mozilla.com:8018/builders/release-mozilla-beta-linux64_standalone_repack/builds/3
http://dev-master1.srv.releng.scl3.mozilla.com:8018/builders/release-mozilla-beta-linux64_repack_1%2F10/builds/5
http://dev-master1.srv.releng.scl3.mozilla.com:8018/builders/release-mozilla-beta-macosx64_repack_1%2F10/builds/0

And Balrog blobs are here:
https://aus4-admin-dev.allizom.org/releases/Firefox-29.0b4-build1/data
https://aus4-admin-dev.allizom.org/releases/Thunderbird-28.0b1-build1/data

Changes since the last patch:
* Address review comments
* Fix checksums file parsing on Windows (need --no-print-directory there, because pymake prints by default, I guess)
* Don't purge oauth.txt!
Attachment #8401270 - Flags: review?(rail)
Comment on attachment 8401265 [details] [diff] [review]
buildbotcustom changes, tested

Review of attachment 8401265 [details] [diff] [review]:
-----------------------------------------------------------------

::: process/release.py
@@ +72,5 @@
>          'base_clobber_url', branchConfig['base_clobber_url'])
> +    balrog_api_root=releaseConfig.get('balrog_api_root',
> +        branchConfig.get('balrog_api_root', None))
> +    balrog_username=releaseConfig.get('balrog_username',
> +        branchConfig.get('balrog_username', None))

A nit. Can you add spaces around = ?
Attachment #8401265 - Flags: review?(rail) → review+
Comment on attachment 8401270 [details] [diff] [review]
tested version of tools patch

Review of attachment 8401270 [details] [diff] [review]:
-----------------------------------------------------------------

sheeep it!
Attachment #8401270 - Flags: review?(rail) → review+
(Assignee)

Comment 10

3 years ago
Comment on attachment 8401265 [details] [diff] [review]
buildbotcustom changes, tested

Landed. Reconfig coming soon, in time for 29.0b6.
Attachment #8401265 - Flags: checked-in+
(Assignee)

Updated

3 years ago
Attachment #8401270 - Flags: checked-in+
(Assignee)

Comment 11

3 years ago
in production
We'll need to add a retry around the submissions, particularly for locales. We hit this error about 15 times in 29.0b6, sometimes one a repack job, sometimes several:

Starting new HTTPS connection (1): aus4-admin.mozilla.org
Starting new HTTPS connection (2): aus4-admin.mozilla.org
Caught HTTPError: Couldn't update release: Failed to update row, old_data_version doesn't match current data_version

http://ftp.mozilla.org/pub/mozilla.org/firefox/candidates/29.0b6-candidates/build1/logs/release-mozilla-beta-linux64_repack_9-bm94-build1-build1.txt.gz
(Assignee)

Comment 13

3 years ago
(In reply to Nick Thomas [:nthomas] from comment #12)
> We'll need to add a retry around the submissions, particularly for locales.
> We hit this error about 15 times in 29.0b6, sometimes one a repack job,
> sometimes several:
> 
> Starting new HTTPS connection (1): aus4-admin.mozilla.org
> Starting new HTTPS connection (2): aus4-admin.mozilla.org
> Caught HTTPError: Couldn't update release: Failed to update row,
> old_data_version doesn't match current data_version
> 
> http://ftp.mozilla.org/pub/mozilla.org/firefox/candidates/29.0b6-candidates/
> build1/logs/release-mozilla-beta-linux64_repack_9-bm94-build1-build1.txt.gz

Yikes, sorry about that! This hasn't been an issue for nightlies, so I guess I assumed retries were baked in at a deep level. Looks like those are handled by the submitter being in a RetryingShellCommand though (https://github.com/mozilla/build-buildbotcustom/blob/master/process/factory.py#L1850). Should be an easy fix here.
(Assignee)

Comment 14

3 years ago
Created attachment 8403500 [details] [diff] [review]
retry balrog submission

I gave this a quick test by hand, seems to work:
bash-4.1$ PATH="/tools/buildbot/bin:/usr/local/bin:/usr/lib64/ccache:/bin:/usr/bin:/usr/local/sbin:/usr/sbin:/sbin:/tools/git/bin:/tools/python27/bin:/tools/python27-mercurial/bin:/home/cltbld/bin" python scripts/scripts/l10n/create-release-repacks.py -c mozilla/production_config.py -r mozilla/release-firefox-mozilla-beta.py -b http://hg.mozilla.org/users/bhearsum_mozilla.com/buildbot-configs -t FIREFOX_29_0b4_RELEASE -p linux64 --locale de --stage-ssh-key ffxbld_dsa --stage-server dev-stage01.srv.releng.scl3.mozilla.com --stage-username ffxbld --hghost hg.mozilla.org --compare-locales-repo-path build/compare-locales --generate-partials --tooltool-manifest browser/config/tooltool-manifests/linux64/releng.manifest --tooltool-script /builds/tooltool.py --tooltool-url http://runtime-binaries.pvt.build.mozilla.org/tooltool --balrog-api-root https://aus4-admin-dev.allizom.org --balrog-username=stage-ffxbld --credentials-file BuildSlaves.py --buildid=20140101010101

<snip>

retry: Calling <bound method ReleaseSubmitter.run of <balrog.submitter.cli.ReleaseSubmitter object at 0x170cb10>> with args: (), kwargs: {'completeMarSize': 35341442, 'buildID': '20140101010101', 'completeMarHash': 'a180d02202f986165c68dfe6ee50839302fb1994b17bdad65e4bd97da6c2935f8230d5465b4ffe58ecba4b530691d1830d31cf71cf46c7a962c419e4618a20dc', 'productName': 'Firefox', 'hashFunction': 'sha512', 'locale': 'de', 'build_number': 1, 'extVersion': '29.0', 'appVersion': '29.0', 'platform': 'linux64', 'version': '29.0b4'}, attempt #1
Starting new HTTPS connection (1): aus4-admin-dev.allizom.org
Starting new HTTPS connection (2): aus4-admin-dev.allizom.org
Attachment #8403500 - Flags: review?(rail)
Attachment #8403500 - Flags: review?(rail) → review+
(Assignee)

Updated

3 years ago
Attachment #8403500 - Flags: checked-in+
(Assignee)

Comment 15

3 years ago
No issues with repacks in 29.0b7. I think we're done here.
Status: NEW → RESOLVED
Last Resolved: 3 years ago
Resolution: --- → FIXED
Looks like this broke ESR.
Status: RESOLVED → REOPENED
Resolution: FIXED → ---
The following tracebacks were detected during repacks:
el:
Traceback (most recent call last):
  File "/builds/slave/rel-m-esr24-osx64_rpk_3-000000/scripts/scripts/l10n/create-release-repacks.py", line 128, in createRepacks
    checksums = parseChecksumsFile(open(checksums_file).read())
IOError: [Errno 2] No such file or directory: 'dist/mac/el/firefox-24.5.0esr.checksums'
Hmm, absObjdir at http://hg.mozilla.org/build/tools/rev/349972686306#l2.53 should be absolute, but it doesn't look like we make it absolute anywhere.
Reading the code tells me that we should upload everything we needed, we failed to submit updates to Balrog only. I'm going to proceed and mark all all l10n repacks ready to unblock the rest.
(Assignee)

Comment 20

3 years ago
(In reply to Rail Aliiev [:rail] from comment #18)
> Hmm, absObjdir at http://hg.mozilla.org/build/tools/rev/349972686306#l2.53
> should be absolute, but it doesn't look like we make it absolute anywhere.

Good catch, I think you're right...the variable name certainly says that it's _supposed_ to be abs, as well as the fact that normpath is used in repackLocale. I'll test a fix for this soon.
(Assignee)

Comment 21

3 years ago
Fixing the ESR bustage is high on my priority list, but I got pulled into even more important b2g work. This will definitely be fixed before the next scheduled release, but if have a chemspill or respin in the near future, this may still be an issue for it.
(Assignee)

Comment 22

3 years ago
Created attachment 8421681 [details] [diff] [review]
balrog-esr-repack-tools-3.diff

So, this problem was even weirder than I thought. I found a few issues:
1) I was joining absObjdir and relative_checksums. This means that we were coming up with paths like /builds/slave/dir/dist/... - I have no idea how this worked for non-ESR, except maybe related to the second point:
2) There was a leading single quote in the echoed checksums file.

The fix is a bit weird too, here's a summary:
* Actually make absObjdir an abspath.
* Adjust how absSourceRepoPath is calculated, because the existing way doesn't work with abspaths on Windows.
* Calculate checksums path based on localeSrcDir (the same directory we run make echo-variable from) to get the correct path to it.
* Remove dead code (localeSrcDir being passed to l10nRepackPrep)
* Print exceptions per locale, in addition to the summary at the end. Makes debugging easier/faster.3

Here's blobs from the successful submissions:
https://aus4-admin-dev.allizom.org/releases/Firefox-30.0b3-build1/data
https://aus4-admin-dev.allizom.org/releases/Thunderbird-30.0b1-build2/data
https://aus4-admin-dev.allizom.org/releases/Firefox-24.5.0esr-build1/data
https://aus4-admin-dev.allizom.org/releases/Thunderbird-24.5.0-build1/data

If you'd like to see the output from some runs, I can link you to my dev-master, too.
Attachment #8421681 - Flags: review?(rail)
Attachment #8421681 - Flags: review?(rail) → review+
(Assignee)

Updated

3 years ago
Attachment #8421681 - Flags: checked-in+
(Assignee)

Comment 23

3 years ago
OK, this should be fixed for good now.
Status: REOPENED → RESOLVED
Last Resolved: 3 years ago3 years ago
Resolution: --- → FIXED
You need to log in before you can comment on or make changes to this bug.