Closed Bug 1250629 Opened 4 years ago Closed 4 years ago

Pass revision and en-US url to l10n repack jobs

Categories

(Release Engineering :: General, defect)

defect
Not set

Tracking

(Not tracked)

RESOLVED FIXED

People

(Reporter: catlee, Assigned: nthomas)

References

(Depends on 1 open bug)

Details

Attachments

(3 files)

We need to update the l10n schedulers to pass the proper revision and URLs for the en-US builds.
Blocks: 848284
Depends on: 1250626
We triggering l10n we pass this a sourcestamp like this:
   Branch     mozilla-central
   Revision   ab0044bfa1df
and revision should be picked up after the work in bug 848284.

catlee and I have discussed pass on the packageUrl property from the en-US build at 
  http://hg.mozilla.org/build/buildbotcustom/file/default/misc.py#l805
then teaching desktop_l10n.py to look for that at
  http://hg.mozilla.org/mozilla-central/file/default/testing/mozharness/scripts/desktop_l10n.py#l351

One minor problem with this approach is that packageUrl points into tinderbox-builds like this example:
  http://archive.mozilla.org/pub/firefox/tinderbox-builds/mozilla-central-linux/1461751335/firefox-49.0a1.en-US.linux-i686.tar.bz2
rather than into a dated directory like /pub/firefox/nightly/2016/04/.... This is completely dependent on the order post_upload spits out the urls for the files it has processed, with the last line matching the package 'winning'. Rather fragile, but triggering is likely to get redone as part of bug 1265595 so perhaps we'd end up pointing at the taskcluster index anyway.
I don't think the specific URL matters, as long as it's not pointing to a 'latest' URL that's subject to being overwritten by subsequent jobs. As long as the URL is valid until the repacks are finished, I think that's fine.
Agreed, though I'd extend the durability of the passed URL to retriggers of the specific job. As long as we're talking days, I doubt that matters, though.
Assignee: nobody → nthomas
Comment on attachment 8753205 [details]
MozReview Request: Bug 1250629 - forward packageUrl property from en-US build to triggered l10n jobs, r?catlee

https://reviewboard.mozilla.org/r/53072/#review50006
Attachment #8753205 - Flags: review?(catlee) → review+
Attachment #8753196 - Flags: review?(catlee) → review+
Comment on attachment 8753196 [details]
MozReview Request: Bug 1250629 - desktop l10n script should accept a packageUrl property which points to en-US build, r?catlee

https://reviewboard.mozilla.org/r/53068/#review50008
Keywords: leave-open
Comment on attachment 8753205 [details]
MozReview Request: Bug 1250629 - forward packageUrl property from en-US build to triggered l10n jobs, r?catlee

https://hg.mozilla.org/build/buildbotcustom/rev/156ec2bc0a8f
https://hg.mozilla.org/build/buildbotcustom/rev/d3c55d9b6b9f
Attachment #8753205 - Flags: checked-in+
In today's nightlies we have these properties for win64:
Name		Value		Source
buildid		20160518030234	buildIDSchedFunc (in triggering build)
builduid		f769d09b003647779edb2f3bf3073c37	buildUIDSchedFunc (in triggering build)
packageUrl	http://archive.mozilla.org/pub/firefox/tinderbox-builds/mozilla-central-win64/1463565754/firefox-49.0a1.en-US.win64.zip	SetProperty Step (in triggering build)

plus branch and revision set on the sourcestamp. 9a6699252704 will be in the next set of nightlies.
This breaks windows because of unicode...we end up passing EN_US_BINARY_URL in the environment as a unicode string, and then subprocess complains:

http://archive.mozilla.org/pub/firefox/nightly/2016/05/2016-05-19-03-02-32-mozilla-central-l10n/mozilla-central-win32-l10n-nightly-1-unknown-bm74-build1-build5.txt.gz

06:56:00     INFO -  'EN_US_BINARY_URL': u'http://archive.mozilla.org/pub/firefox/tinderbox-builds/mozilla-central-win32/1463652152',
...
06:56:00     INFO - #####
06:56:00     INFO - ##### Finished setup step (failed)
06:56:00     INFO - #####
06:56:01    FATAL - Uncaught exception: Traceback (most recent call last):
06:56:01    FATAL -   File "c:\builds\moz2_slave\m-cen-w32-l10n-ntly-1-00000000\scripts\mozharness\base\script.py", line 1768, in run
06:56:01    FATAL -     self.run_action(action)
06:56:01    FATAL -   File "c:\builds\moz2_slave\m-cen-w32-l10n-ntly-1-00000000\scripts\mozharness\base\script.py", line 1707, in run_action
06:56:01    FATAL -     self._possibly_run_method(method_name, error_if_missing=True)
06:56:01    FATAL -   File "c:\builds\moz2_slave\m-cen-w32-l10n-ntly-1-00000000\scripts\mozharness\base\script.py", line 1647, in _possibly_run_method
06:56:01    FATAL -     return getattr(self, method_name)()
06:56:01    FATAL -   File "scripts/scripts/desktop_l10n.py", line 647, in setup
06:56:01    FATAL -     self._mach_configure()
06:56:01    FATAL -   File "scripts/scripts/desktop_l10n.py", line 716, in _mach_configure
06:56:01    FATAL -     return self._mach(target=target, env=env)
06:56:01    FATAL -   File "scripts/scripts/desktop_l10n.py", line 710, in _mach
06:56:01    FATAL -     output_parser=None)
06:56:01    FATAL -   File "c:\builds\moz2_slave\m-cen-w32-l10n-ntly-1-00000000\scripts\mozharness\base\script.py", line 1172, in run_command
06:56:01    FATAL -     cwd=cwd, stderr=subprocess.STDOUT, env=env)
06:56:01    FATAL -   File "c:\mozilla-build\python27\lib\subprocess.py", line 711, in __init__
06:56:01    FATAL -     errread, errwrite)
06:56:01    FATAL -   File "c:\mozilla-build\python27\lib\subprocess.py", line 948, in _execute_child
06:56:01    FATAL -     startupinfo)
06:56:01    FATAL - TypeError: environment can only contain strings
06:56:01    FATAL - Running post_fatal callback...
06:56:01    FATAL - Exiting -1
06:56:01     INFO - Running post-run listener: copy_logs_to_upload_dir
06:56:01     INFO - Copying logs to upload dir...
Comment on attachment 8754470 [details]
MozReview Request: Bug 1250629 - stringify en_us_package_url r=nthomas

https://reviewboard.mozilla.org/r/53984/#review50704

Stealing the review, as this is simple enough.
Attachment #8754470 - Flags: review+
Attachment #8754470 - Flags: review?(nthomas) → review+
Comment on attachment 8754470 [details]
MozReview Request: Bug 1250629 - stringify en_us_package_url r=nthomas

https://reviewboard.mozilla.org/r/53984/#review51628
From a en-US nightly:

10:47:19     INFO - Overriding en_us_binary_url with http://archive.mozilla.org/pub/firefox/tinderbox-builds/mozilla-central-win32/1464100634
Status: NEW → RESOLVED
Closed: 4 years ago
Resolution: --- → FIXED
The change on that bug actually busted our testing of localized builds in mozmill-ci, which means we were running en-US builds instead of the locale repacks since May 18th when the first patch on that bug got landed. For details see https://github.com/mozilla/mozmill-ci/issues/796.
Removing leave-open keyword from resolved bugs, per :sylvestre.
Keywords: leave-open
Component: General Automation → General
You need to log in before you can comment on or make changes to this bug.