1464026 - Tracking bug for 2018-06-26 migration work

Status: RESOLVED FIXED

(Release Engineering :: Release Requests, enhancement)

Comment 1

[Mihai Tabara [:mtabara]⌚️GMT]

Adding bug 1415233 as dep to have this in mind when we're attempting the mergeduty work for 61.

Comment 2

[Mihai Tabara [:mtabara]⌚️GMT]

Dry-runs take for ever in the new instance. Likely puppet runs are wiping some of the information. More details in bug 1469284.
Anyway, when trying to push the beta -> release merge, it failed with the following error:

11:15:12     INFO -  Traceback (most recent call last):
11:15:12     INFO -    File "/tools/python27-mercurial/lib/python2.7/site-packages/mercurial/scmutil.py", line 154, in callcatch
11:15:12     INFO -      return func()
11:15:12     INFO -    File "/tools/python27-mercurial/lib/python2.7/site-packages/mercurial/dispatch.py", line 303, in _runcatchfunc
11:15:12     INFO -      return _dispatch(req)
11:15:12     INFO -    File "/tools/python27-mercurial/lib/python2.7/site-packages/mercurial/dispatch.py", line 940, in _dispatch
11:15:12     INFO -      cmdpats, cmdoptions)
11:15:12     INFO -    File "/tools/python27-mercurial/lib/python2.7/site-packages/mercurial/dispatch.py", line 701, in runcommand
11:15:12     INFO -      ret = _runcommand(ui, options, cmd, d)
11:15:12     INFO -    File "/tools/python27-mercurial/lib/python2.7/site-packages/mercurial/dispatch.py", line 948, in _runcommand
11:15:12     INFO -      return cmdfunc()
11:15:12     INFO -    File "/tools/python27-mercurial/lib/python2.7/site-packages/mercurial/dispatch.py", line 937, in <lambda>
11:15:12     INFO -      d = lambda: util.checksignature(func)(ui, *args, **strcmdopt)
11:15:12     INFO -    File "/tools/python27-mercurial/lib/python2.7/site-packages/mercurial/util.py", line 1183, in check
11:15:12     INFO -      return func(*args, **kwargs)
11:15:12     INFO -    File "/tools/python27-mercurial/lib/python2.7/site-packages/mercurial/util.py", line 1183, in check
11:15:12     INFO -      return func(*args, **kwargs)
11:15:12     INFO -    File "/tools/python27-mercurial/lib/python2.7/site-packages/hgext/mq.py", line 3583, in mqcommand
11:15:12     INFO -      return orig(ui, repo, *args, **kwargs)
11:15:12     INFO -    File "/tools/python27-mercurial/lib/python2.7/site-packages/mercurial/util.py", line 1183, in check
11:15:12     INFO -      return func(*args, **kwargs)
11:15:12     INFO -    File "/tools/python27-mercurial/lib/python2.7/site-packages/mercurial/commands.py", line 4160, in push
11:15:12     INFO -      opargs=opargs)
11:15:12     INFO -    File "/tools/python27-mercurial/lib/python2.7/site-packages/mercurial/exchange.py", line 473, in push
11:15:12     INFO -      _pushbundle2(pushop)
11:15:12     INFO -    File "/tools/python27-mercurial/lib/python2.7/site-packages/mercurial/exchange.py", line 987, in _pushbundle2
11:15:12     INFO -      raise error.Abort(_('push failed on remote'))
11:15:12     INFO -  Abort: push failed on remote
11:15:12    ERROR -  abort: push failed on remote
11:15:12    ERROR -  Automation Error: hg not responding
11:15:13    ERROR - Return code: 255
11:15:13    FATAL - Push failed!  If there was a push race, try rerunning
11:15:13    FATAL - the script (--clean-repos --pull --migrate).  The second run will be faster.
11:15:13    FATAL - Running post_fatal callback...
11:15:13    FATAL - Exiting -1
11:15:13     INFO - [mozharness: 2018-06-18 18:15:13.204640Z] Finished push step (failed)
11:15:13     INFO - Running post-run listener: copy_logs_to_upload_dir
11:15:13     INFO - Copying logs to upload dir...


Need to debug this or rerun. I'm likely at this point to ramp-up old instance and work on there. It was much faster.

Comment 3

[Jordan Lund (:jlund)]

(In reply to Mihai Tabara [:mtabara]⌚️GMT from comment #1)
> Adding bug 1415233 as dep to have this in mind when we're attempting the
> mergeduty work for 61.

bit us again! we should add this to docs

Comment 4

[Jordan Lund (:jlund)]

update: used old instance to push beta and release

current issues:
* pretxnclose.vcsreplicator hook failed bug 1415233. connor came to the rescue. We need to do this along with the ftl hooks before pushing to each repo with large commit sets
* bump_esr only handles esr52 (not esr60). I modified the bump_esr mh config to point to esr60 after doing esr52
* esr still tries to push over http. needed to modify esr repo's hgrc to include default-push via ssh:
   * mentioned in https://github.com/mozilla-releng/releasewarrior-2.0/blob/9b226e7b8207bf84be153b2063434c8b02d4107a/docs/mergeduty/howto.md#bump-esr-version

Comment 5

[Jordan Lund (:jlund)]

all done for today's work.

* esrs bumped
* l10n bumped - I temporarily removed "DONTBUILD" from commit message as a work around until 1469423 is resolved
* m-r merged
* m-b merged

next week:

* bump m-c
* update bouncer aliases
* update wiki page gecko versions

Comment 6

[Jordan Lund (:jlund)]

turns out l10n bumper run didn't work as expected:

RyanVM> lizzard: wait, did the l10n bump push never happen?
18:56:17 jlund → jlund|away
18:56:53 
<•jlund|away> Jordan Lund RyanVM: I ran the bumper script manually and it didn’t fail?
18:57:24 
<lizzard> I thought jlund did that earlier?
18:57:35 ah
18:58:15 
<RyanVM> i don't see any resulting push to beta changing all the revisions from default to real hashes
18:58:16 https://hg.mozilla.org/releases/mozilla-beta/file/default/browser/locales/l10n-changesets.json
19:05:21 
<•aki> doh
19:07:38 running
<•aki> i just pushed l10n changesets to beta manually
19:30:11 
<RyanVM> thanks!
<•jlund> Jordan Lund thanks aki-away
22:08:58 be good to know what is missing from l10n bump docs that led to us not having what we want on m-b: https://github.com/mozilla-releng/releasewarrior-2.0/blob/master/docs/mergeduty/howto.md#run-the-l10n-bumper

need to update docs

Comment 7

[Mihai Tabara [:mtabara]⌚️GMT]

Successfully used the new puppetized instance to perfrom second part of mergeduty (bump central + tagging).

Note to self:
* I logged in directly as buildduty via `ssh +A [e-mail]`
* I copied over several times the bkp ssh config over to ~/.ssh/config to make sure it's using

Host hg.mozilla.org
    IdentityFile ~/mtabara_rsa.pub
    User [e-mail]

* I copied over behforehand my key on the remote instance in the ~ folder

Comment 8

[Mihai Tabara [:mtabara]⌚️GMT]

As of now, we're up-to-date to https://github.com/mozilla-releng/releasewarrior-2.0/blob/master/docs/mergeduty/howto.md#bump-bouncer-versions. Pending green nightlies we can trigger follow-up with bouncer entries.

Comment 9

[Jordan Lund (:jlund)]

(In reply to Mihai Tabara [:mtabara]⌚️GMT from comment #8)
> As of now, we're up-to-date to
> https://github.com/mozilla-releng/releasewarrior-2.0/blob/master/docs/
> mergeduty/howto.md#bump-bouncer-versions. Pending green nightlies we can
> trigger follow-up with bouncer entries.

bouncer aliases updated. note, we don't need to update the bouncer locations for stub aliases anymore: firefox-nightly-stub and
firefox-nightly-stub-l10n

e.g. the paths in bouncer for those assume the following simplified format without version in the string: /firefox/nightly/latest-mozilla-central-l10n/Firefox%20Installer.:lang.exe

This lines up with what ends up in firefox/nightly/latest-mozilla-central: https://archive.mozilla.org/pub/firefox/nightly/latest-mozilla-central/Firefox%20Installer.en-US.exe

I'll remove that from those docs

Comment 10

[Mihai Tabara [:mtabara]⌚️GMT]

Note to self: we also should consider stopping the mergeduty instances to save some $$$ in AWS.

Comment 11

[Mihai Tabara [:mtabara]⌚️GMT]

Note to self: might be handy to add support for mergeduty steps in warrior?

Comment 12

[Mihai Tabara [:mtabara]⌚️GMT]

Note to self: if we add mergeduty steps docs somwhere in warrior or in docs - we should add reviewing/merging/landing stuff like https://github.com/mozilla-releng/ship-it/pull/225 too. Right now it's unclear how's following up with that patch and when do we land it.

Comment 13

[Mihai Tabara [:mtabara]⌚️GMT]

This dates back to 61.0 cycle which is now history. I think we can close this.