1337027 - l10n process started before tag it references lands!

Status: RESOLVED FIXED

(Release Engineering :: Release Automation: L10N, defect, P4)

Description

[Hal Wine [:hwine] use NI!]

This caused a failure in XH repack of FENNEC_51_0_2_RELEASE build.

HG pull requested at 1007 UTC, tag only pushed to HG at 1021 UTC

From the failure email thread:

From: Rail Aliiev 
Date: Mon, Feb 6, 2017 at 2:44 AM PT
Subject: Re: [release] Fennec 51.0.2 build1: build step failed on android-api-15


This one looks like an intermittent hg issue, rerunning.


On 02:20 PT, Mon, 06 Feb, 

    android-api-15 build step failed for Fennec 51.0.2 build1

    Status: failure

    Full details are available at:
    http://buildbot-master70.bb.releng.use1.mozilla.com:8001/builders/release-mozilla-release-android-api-15_build/builds/0

    -buildbot

From the xh repo pushlog:
https://hg.mozilla.org/releases/l10n/mozilla-release/xh/pushloghtml?startdate=2017-02-06+10%3A00&enddate=2017-02-06+10%3A30
ffxbld
Mon Feb 06 10:21:17 2017 +0000	aea6ce822a35	ffxbld — Added FENNEC_51_0_2_RELEASE FENNEC_51_0_2_BUILD1 tag(s) for changeset f587b401f8fc. DONTBUILD CLOSED TREE a=release MOBILE5102_2017020604_RELBRANCH tip 

From the log (note buildbot timestamps in UTC-0500):

05:07:59     INFO - [mozharness: 2017-02-06 10:07:59.538927Z] Skipping clobber step.
05:07:59     INFO - [mozharness: 2017-02-06 10:07:59.539142Z] Skipping pull-build-source step.
05:07:59     INFO - [mozharness: 2017-02-06 10:07:59.539352Z] Running pull-locale-source step.
05:07:59     INFO - Running main action method: pull_locale_source
...
05:08:17     INFO - Running command: ['hg', '--config', 'ui.merge=internal:merge', '--config', 'extensions.robustcheckout=/builds/slave/rel-m-rel-and-api-15_bld-00000/mozharness/external_tools/robustcheckout.py', 'robustcheckout', u'https://hg.mozilla.org/releases/l10n/mozilla-release/xh', u'xh', '--sharebase', u'/builds/hg-shared', '--branch', 'FENNEC_51_0_2_RELEASE']
05:08:17     INFO - Copy/paste: hg --config ui.merge=internal:merge --config extensions.robustcheckout=/builds/slave/rel-m-rel-and-api-15_bld-00000/mozharness/external_tools/robustcheckout.py robustcheckout https://hg.mozilla.org/releases/l10n/mozilla-release/xh xh --sharebase /builds/hg-shared --branch FENNEC_51_0_2_RELEASE
05:08:18     INFO -  ensuring https://hg.mozilla.org/releases/l10n/mozilla-release/xh@FENNEC_51_0_2_RELEASE is available at xh
05:08:18     INFO -  warning: connecting to hg.mozilla.org using legacy security technology (TLS 1.0); see https://mercurial-scm.org/wiki/SecureConnections for more info
05:08:19     INFO -  (sharing from new pooled repository 29faceccbba97783b8518070b1e0df99e6260886)
05:08:19     INFO -  warning: connecting to hg.mozilla.org using legacy security technology (TLS 1.0); see https://mercurial-scm.org/wiki/SecureConnections for more info
05:08:22     INFO -  requesting all changes
05:08:22     INFO -  adding changesets
05:08:22     INFO -  adding manifests
05:08:22     INFO -  adding file changes
05:08:22     INFO -  added 1033 changesets with 5748 changes to 829 files (+560 heads)
05:08:22     INFO -  searching for changes
05:08:22     INFO -  no changes found
05:08:22     INFO -  (pulling to obtain FENNEC_51_0_2_RELEASE)
05:08:22     INFO -  warning: connecting to hg.mozilla.org using legacy security technology (TLS 1.0); see https://mercurial-scm.org/wiki/SecureConnections for more info
05:08:23     INFO -  Traceback (most recent call last):
05:08:23     INFO -    File "/tools/python27-mercurial/lib/python2.7/site-packages/mercurial/dispatch.py", line 204, in _runcatch
05:08:23     INFO -      return _dispatch(req)
05:08:23     INFO -    File "/tools/python27-mercurial/lib/python2.7/site-packages/mercurial/dispatch.py", line 880, in _dispatch
05:08:23     INFO -      cmdpats, cmdoptions)
05:08:23     INFO -    File "/tools/python27-mercurial/lib/python2.7/site-packages/mercurial/dispatch.py", line 637, in runcommand
05:08:23     INFO -      ret = _runcommand(ui, options, cmd, d)
05:08:23     INFO -    File "/tools/python27-mercurial/lib/python2.7/site-packages/mercurial/dispatch.py", line 1010, in _runcommand
05:08:23     INFO -      return checkargs()
05:08:23     INFO -    File "/tools/python27-mercurial/lib/python2.7/site-packages/mercurial/dispatch.py", line 971, in checkargs
05:08:23     INFO -      return cmdfunc()
05:08:23     INFO -    File "/tools/python27-mercurial/lib/python2.7/site-packages/mercurial/dispatch.py", line 877, in <lambda>
05:08:23     INFO -      d = lambda: util.checksignature(func)(ui, *args, **cmdoptions)
05:08:23     INFO -    File "/tools/python27-mercurial/lib/python2.7/site-packages/mercurial/util.py", line 1036, in check
05:08:23     INFO -      return func(*args, **kwargs)
05:08:23     INFO -    File "/builds/slave/rel-m-rel-and-api-15_bld-00000/mozharness/external_tools/robustcheckout.py", line 149, in robustcheckout
05:08:23     INFO -      sharebase)
05:08:23     INFO -    File "/builds/slave/rel-m-rel-and-api-15_bld-00000/mozharness/external_tools/robustcheckout.py", line 292, in _docheckout
05:08:23     INFO -      return handlerepoerror(e)
05:08:23     INFO -    File "/builds/slave/rel-m-rel-and-api-15_bld-00000/mozharness/external_tools/robustcheckout.py", line 271, in _docheckout
05:08:23     INFO -      pullrevs = [remote.lookup(revision or branch)]
05:08:23     INFO -    File "/tools/python27-mercurial/lib/python2.7/site-packages/mercurial/peer.py", line 106, in plain
05:08:23     INFO -      return next(batchable)
05:08:23     INFO -    File "/tools/python27-mercurial/lib/python2.7/site-packages/mercurial/wireproto.py", line 264, in lookup
05:08:23     INFO -      self._abort(error.RepoError(data))
05:08:23     INFO -    File "/tools/python27-mercurial/lib/python2.7/site-packages/mercurial/httppeer.py", line 277, in _abort
05:08:23     INFO -      raise exception
05:08:23     INFO -  RepoError: unknown revision 'FENNEC_51_0_2_RELEASE'
05:08:23     INFO -  abort: unknown revision 'FENNEC_51_0_2_RELEASE'!
05:08:23    ERROR - Return code: 255


From pushlog for

Comment 1

[Gregory Szorc [:gps]]

See the blue box at https://mozilla-version-control-tools.readthedocs.io/en/latest/hgmo/notifications.html#common-properties-of-notifications then consider changing the mechanism for scheduling jobs to use Pulse or SNS as its primary mechanism instead of polling the pushlog.

Comment 2

[Hal Wine [:hwine] use NI!]

Er, we don't poll the pushlog -- I just put the pushlog data in there to show when that tag arrived on the server. The tag step should complete well before the pull (and normally does). This isn't a replication issue, it's a scheduling glitch.

Comment 3

[Rail Aliiev [:rail]]

Apparently rerunning an ARM android build is bad, we regenerate the version, which based on the timestamp and assume that x86's version is "greater" than ARM (Play Store related).

Comment 4

[Aki Sasaki (not active)]

I think the behavior here has changed - we pin revisions in l10n-changesets.json. We shouldn't see this again.