1250629 - Pass revision and en-US url to l10n repack jobs

Status: RESOLVED FIXED

(Release Engineering :: General, defect)

Description

[Chris AtLee [:catlee]]

We need to update the l10n schedulers to pass the proper revision and URLs for the en-US builds.

Comment 1

[Nick Thomas [:nthomas] (UTC+12)]

We triggering l10n we pass this a sourcestamp like this:
   Branch     mozilla-central
   Revision   ab0044bfa1df
and revision should be picked up after the work in bug 848284.

catlee and I have discussed pass on the packageUrl property from the en-US build at 
  http://hg.mozilla.org/build/buildbotcustom/file/default/misc.py#l805
then teaching desktop_l10n.py to look for that at
  http://hg.mozilla.org/mozilla-central/file/default/testing/mozharness/scripts/desktop_l10n.py#l351

One minor problem with this approach is that packageUrl points into tinderbox-builds like this example:
  http://archive.mozilla.org/pub/firefox/tinderbox-builds/mozilla-central-linux/1461751335/firefox-49.0a1.en-US.linux-i686.tar.bz2
rather than into a dated directory like /pub/firefox/nightly/2016/04/.... This is completely dependent on the order post_upload spits out the urls for the files it has processed, with the last line matching the package 'winning'. Rather fragile, but triggering is likely to get redone as part of bug 1265595 so perhaps we'd end up pointing at the taskcluster index anyway.

Comment 2

[Chris AtLee [:catlee]]

I don't think the specific URL matters, as long as it's not pointing to a 'latest' URL that's subject to being overwritten by subsequent jobs. As long as the URL is valid until the repacks are finished, I think that's fine.

Comment 3

[Axel Hecht [:Pike]]

Agreed, though I'd extend the durability of the passed URL to retriggers of the specific job. As long as we're talking days, I doubt that matters, though.

Comment 4

[Nick Thomas [:nthomas] (UTC+12)]

Attachment: MozReview Request: Bug 1250629 - desktop l10n script should accept a packageUrl property which points to en-US build, r?catlee

Review commit: https://reviewboard.mozilla.org/r/53068/diff/#index_header
See other reviews: https://reviewboard.mozilla.org/r/53068/

Comment 5

[Nick Thomas [:nthomas] (UTC+12)]

Attachment: MozReview Request: Bug 1250629 - forward packageUrl property from en-US build to triggered l10n jobs, r?catlee

Review commit: https://reviewboard.mozilla.org/r/53072/diff/#index_header
See other reviews: https://reviewboard.mozilla.org/r/53072/

Comment 6

[Chris AtLee [:catlee]]

Comment on attachment 8753205 [details]
MozReview Request: Bug 1250629 - forward packageUrl property from en-US build to triggered l10n jobs, r?catlee

https://reviewboard.mozilla.org/r/53072/#review50006

Comment 7

[Chris AtLee [:catlee]]

Comment on attachment 8753196 [details]
MozReview Request: Bug 1250629 - desktop l10n script should accept a packageUrl property which points to en-US build, r?catlee

https://reviewboard.mozilla.org/r/53068/#review50008

Comment 8

[Nick Thomas [:nthomas] (UTC+12)]

Comment on attachment 8753205 [details]
MozReview Request: Bug 1250629 - forward packageUrl property from en-US build to triggered l10n jobs, r?catlee

https://hg.mozilla.org/build/buildbotcustom/rev/156ec2bc0a8f
https://hg.mozilla.org/build/buildbotcustom/rev/d3c55d9b6b9f

Comment 9

[Pulsebot]

https://hg.mozilla.org/integration/mozilla-inbound/rev/9a6699252704

Comment 10

[Ryan VanderMeulen [:RyanVM]]

https://hg.mozilla.org/mozilla-central/rev/9a6699252704

Comment 11

[Nick Thomas [:nthomas] (UTC+12)]

In today's nightlies we have these properties for win64:
Name		Value		Source
buildid		20160518030234	buildIDSchedFunc (in triggering build)
builduid		f769d09b003647779edb2f3bf3073c37	buildUIDSchedFunc (in triggering build)
packageUrl	http://archive.mozilla.org/pub/firefox/tinderbox-builds/mozilla-central-win64/1463565754/firefox-49.0a1.en-US.win64.zip	SetProperty Step (in triggering build)

plus branch and revision set on the sourcestamp. 9a6699252704 will be in the next set of nightlies.

Comment 12

[Chris AtLee [:catlee]]

This breaks windows because of unicode...we end up passing EN_US_BINARY_URL in the environment as a unicode string, and then subprocess complains:

http://archive.mozilla.org/pub/firefox/nightly/2016/05/2016-05-19-03-02-32-mozilla-central-l10n/mozilla-central-win32-l10n-nightly-1-unknown-bm74-build1-build5.txt.gz

06:56:00     INFO -  'EN_US_BINARY_URL': u'http://archive.mozilla.org/pub/firefox/tinderbox-builds/mozilla-central-win32/1463652152',
...
06:56:00     INFO - #####
06:56:00     INFO - ##### Finished setup step (failed)
06:56:00     INFO - #####
06:56:01    FATAL - Uncaught exception: Traceback (most recent call last):
06:56:01    FATAL -   File "c:\builds\moz2_slave\m-cen-w32-l10n-ntly-1-00000000\scripts\mozharness\base\script.py", line 1768, in run
06:56:01    FATAL -     self.run_action(action)
06:56:01    FATAL -   File "c:\builds\moz2_slave\m-cen-w32-l10n-ntly-1-00000000\scripts\mozharness\base\script.py", line 1707, in run_action
06:56:01    FATAL -     self._possibly_run_method(method_name, error_if_missing=True)
06:56:01    FATAL -   File "c:\builds\moz2_slave\m-cen-w32-l10n-ntly-1-00000000\scripts\mozharness\base\script.py", line 1647, in _possibly_run_method
06:56:01    FATAL -     return getattr(self, method_name)()
06:56:01    FATAL -   File "scripts/scripts/desktop_l10n.py", line 647, in setup
06:56:01    FATAL -     self._mach_configure()
06:56:01    FATAL -   File "scripts/scripts/desktop_l10n.py", line 716, in _mach_configure
06:56:01    FATAL -     return self._mach(target=target, env=env)
06:56:01    FATAL -   File "scripts/scripts/desktop_l10n.py", line 710, in _mach
06:56:01    FATAL -     output_parser=None)
06:56:01    FATAL -   File "c:\builds\moz2_slave\m-cen-w32-l10n-ntly-1-00000000\scripts\mozharness\base\script.py", line 1172, in run_command
06:56:01    FATAL -     cwd=cwd, stderr=subprocess.STDOUT, env=env)
06:56:01    FATAL -   File "c:\mozilla-build\python27\lib\subprocess.py", line 711, in __init__
06:56:01    FATAL -     errread, errwrite)
06:56:01    FATAL -   File "c:\mozilla-build\python27\lib\subprocess.py", line 948, in _execute_child
06:56:01    FATAL -     startupinfo)
06:56:01    FATAL - TypeError: environment can only contain strings
06:56:01    FATAL - Running post_fatal callback...
06:56:01    FATAL - Exiting -1
06:56:01     INFO - Running post-run listener: copy_logs_to_upload_dir
06:56:01     INFO - Copying logs to upload dir...

Comment 13

[Chris AtLee [:catlee]]

Attachment: MozReview Request: Bug 1250629 - stringify en_us_package_url r=nthomas

Review commit: https://reviewboard.mozilla.org/r/53984/diff/#index_header
See other reviews: https://reviewboard.mozilla.org/r/53984/

Comment 14

[Axel Hecht [:Pike]]

Comment on attachment 8754470 [details]
MozReview Request: Bug 1250629 - stringify en_us_package_url r=nthomas

https://reviewboard.mozilla.org/r/53984/#review50704

Stealing the review, as this is simple enough.

Comment 15

[Pulsebot]

https://hg.mozilla.org/integration/mozilla-inbound/rev/fbdb4502afd5

Comment 16

[Wes Kocher (:KWierso) (Not reading bugmail; email directly if needed)]

https://hg.mozilla.org/mozilla-central/rev/fbdb4502afd5

Comment 17

[Nick Thomas [:nthomas] (UTC+12)]

Comment on attachment 8754470 [details]
MozReview Request: Bug 1250629 - stringify en_us_package_url r=nthomas

https://reviewboard.mozilla.org/r/53984/#review51628

Comment 18

[Nick Thomas [:nthomas] (UTC+12)]

From a en-US nightly:

10:47:19     INFO - Overriding en_us_binary_url with http://archive.mozilla.org/pub/firefox/tinderbox-builds/mozilla-central-win32/1464100634

Comment 19

[Henrik Skupin [:whimboo][⌚️UTC+1]]

The change on that bug actually busted our testing of localized builds in mozmill-ci, which means we were running en-US builds instead of the locale repacks since May 18th when the first patch on that bug got landed. For details see https://github.com/mozilla/mozmill-ci/issues/796.

Comment 20

[Firefox Bug Husbandry Bot]

Removing leave-open keyword from resolved bugs, per :sylvestre.