Closed Bug 1274488 Opened 9 years ago Closed 9 years ago

Android Nightly builds broken because Mercurial 3.1.2 is used

Categories

(Release Engineering :: Applications: MozharnessCore, defect)

Product:
Release Engineering
Component:
Applications: MozharnessCore
Type:
defect
Priority:
Not set
Severity:
normal

Tracking

(firefox49blocking verified)

Status:
RESOLVED FIXED
Tracking Flags:
Tracking Status
firefox49 blocking verified

People

(Reporter: gps, Assigned: nthomas)

References

Details

Attachments

(1 file)

[puppet] Add mozilla-mercurial repo
1.32 KB, patch
dustin
: review+
nthomas
: checked-in+
Details | Diff | Splinter Review
More fallout from bug 1270317. First it was bug 1274436. Now that it is fixed, we found a new bug due to using Mercurial 3.1.2: https://treeherder.mozilla.org/logviewer.html#?job_id=3914824&repo=mozilla-central#L27005 19:19:59 INFO - 22:19:59 INFO - Running command: ['hg', '--config', 'ui.merge=internal:merge', '--version'] 19:19:59 INFO - 22:19:59 INFO - Copy/paste: hg --config ui.merge=internal:merge --version 19:19:59 INFO - 22:19:59 INFO - Mercurial Distributed SCM (version 3.1.2) 19:19:59 INFO - 22:19:59 INFO - (see http://mercurial.selenic.com for more information) 19:19:59 INFO - 22:19:59 INFO - Copyright (C) 2005-2014 Matt Mackall and others 19:19:59 INFO - 22:19:59 INFO - This is free software; see the source for copying conditions. There is NO 19:19:59 INFO - 22:19:59 INFO - warranty; not even for MERCHANTABILITY or FITNESS FOR A PARTICULAR PURPOSE. 19:19:59 INFO - 22:19:59 INFO - Return code: 0 19:19:59 INFO - 22:19:59 INFO - Running command: ['hg', '--config', 'ui.merge=internal:merge', 'debuginstall'] 19:19:59 INFO - 22:19:59 INFO - Copy/paste: hg --config ui.merge=internal:merge debuginstall 19:20:00 INFO - 22:20:00 INFO - checking encoding (ascii)... 19:20:00 INFO - 22:20:00 INFO - checking Python executable (/tools/python27/bin/python2.7) 19:20:00 INFO - 22:20:00 INFO - checking Python version (2.7.7) 19:20:00 INFO - 22:20:00 INFO - checking Python lib (/tools/python27/lib/python2.7)... 19:20:00 INFO - 22:20:00 INFO - checking installed modules (/tools/python27-mercurial/lib/python2.7/site-packages/mercurial)... 19:20:00 INFO - 22:20:00 INFO - checking templates (/tools/python27-mercurial/lib/python2.7/site-packages/mercurial/templates)... 19:20:00 INFO - 22:20:00 INFO - checking commit editor... 19:20:00 INFO - 22:20:00 INFO - No commit editor set and can't find vi in PATH 19:20:00 INFO - 22:20:00 INFO - (specify a commit editor in your configuration file) 19:20:00 INFO - 22:20:00 INFO - checking username... 19:20:00 INFO - 22:20:00 INFO - no problems detected 19:20:00 INFO - 22:20:00 INFO - Return code: 0 19:20:00 INFO - 22:20:00 INFO - Running command: ['hg', '--config', 'ui.merge=internal:merge', '--config', 'extensions.robustcheckout=/builds/slave/m-cen-and-api-15-ntly-00000000/scripts/external_tools/robustcheckout.py', 'robustcheckout', u'https://hg.mozilla.org/l10n-central/cs', u'cs', '--sharebase', u'/builds/hg-shared', '--revision', u'default'] 19:20:00 INFO - 22:20:00 INFO - Copy/paste: hg --config ui.merge=internal:merge --config extensions.robustcheckout=/builds/slave/m-cen-and-api-15-ntly-00000000/scripts/external_tools/robustcheckout.py robustcheckout https://hg.mozilla.org/l10n-central/cs cs --sharebase /builds/hg-shared --revision default 19:20:00 INFO - 22:20:00 INFO - ensuring https://hg.mozilla.org/l10n-central/cs@default is available at cs 19:20:00 INFO - 22:20:00 INFO - Traceback (most recent call last): 19:20:00 INFO - 22:20:00 INFO - File "/tools/python27-mercurial/lib/python2.7/site-packages/mercurial/dispatch.py", line 138, in _runcatch 19:20:00 INFO - 22:20:00 INFO - return _dispatch(req) 19:20:00 INFO - 22:20:00 INFO - File "/tools/python27-mercurial/lib/python2.7/site-packages/mercurial/dispatch.py", line 839, in _dispatch 19:20:00 INFO - 22:20:00 INFO - cmdpats, cmdoptions) 19:20:00 INFO - 22:20:00 INFO - File "/tools/python27-mercurial/lib/python2.7/site-packages/mercurial/dispatch.py", line 600, in runcommand 19:20:00 INFO - 22:20:00 INFO - ret = _runcommand(ui, options, cmd, d) 19:20:00 INFO - 22:20:00 INFO - File "/tools/python27-mercurial/lib/python2.7/site-packages/mercurial/dispatch.py", line 930, in _runcommand 19:20:00 INFO - 22:20:00 INFO - return checkargs() 19:20:00 INFO - 22:20:00 INFO - File "/tools/python27-mercurial/lib/python2.7/site-packages/mercurial/dispatch.py", line 901, in checkargs 19:20:00 INFO - 22:20:00 INFO - return cmdfunc() 19:20:00 INFO - 22:20:00 INFO - File "/tools/python27-mercurial/lib/python2.7/site-packages/mercurial/dispatch.py", line 836, in <lambda> 19:20:00 INFO - 22:20:00 INFO - d = lambda: util.checksignature(func)(ui, *args, **cmdoptions) 19:20:00 INFO - 22:20:00 INFO - File "/tools/python27-mercurial/lib/python2.7/site-packages/mercurial/util.py", line 550, in check 19:20:00 INFO - 22:20:00 INFO - return func(*args, **kwargs) 19:20:00 INFO - 22:20:00 INFO - File "/builds/slave/m-cen-and-api-15-ntly-00000000/scripts/external_tools/robustcheckout.py", line 142, in robustcheckout 19:20:00 INFO - 22:20:00 INFO - sharebase) 19:20:00 INFO - 22:20:00 INFO - File "/builds/slave/m-cen-and-api-15-ntly-00000000/scripts/external_tools/robustcheckout.py", line 213, in _docheckout 19:20:00 INFO - 22:20:00 INFO - makedirs(os.path.dirname(destvfs.base), notindexed=True) 19:20:00 INFO - 22:20:00 INFO - TypeError: ensuredirs() got an unexpected keyword argument 'notindexed' 19:20:00 INFO - 22:20:00 INFO - ** Unknown exception encountered with possibly-broken third-party extension bundleclone 19:20:00 INFO - 22:20:00 INFO - ** which supports versions 3.1 of Mercurial. 19:20:00 INFO - 22:20:00 INFO - ** Please disable bundleclone and try your action again. 19:20:00 INFO - 22:20:00 INFO - ** If that fixes the bug please report it to https://bugzilla.mozilla.org/enter_bug.cgi?product=Developer%20Services&component=Mercurial%3A%20bundleclone 19:20:00 INFO - 22:20:00 INFO - ** Python 2.7.7 (default, Jun 19 2014, 08:59:14) [GCC 4.4.6 20110731 (Red Hat 4.4.6-3)] 19:20:00 INFO - 22:20:00 INFO - ** Mercurial Distributed SCM (version 3.1.2) 19:20:00 INFO - 22:20:00 INFO - ** Extensions loaded: share, rebase, strip, mq, purge, bundleclone, robustcheckout 19:20:00 INFO - 22:20:00 INFO - Traceback (most recent call last): 19:20:00 INFO - 22:20:00 INFO - File "/usr/local/bin/hg", line 43, in <module> 19:20:00 INFO - 22:20:00 INFO - mercurial.dispatch.run() 19:20:00 INFO - 22:20:00 INFO - File "/tools/python27-mercurial/lib/python2.7/site-packages/mercurial/dispatch.py", line 28, in run 19:20:00 INFO - 22:20:00 INFO - sys.exit((dispatch(request(sys.argv[1:])) or 0) & 255) 19:20:00 INFO - 22:20:00 INFO - File "/tools/python27-mercurial/lib/python2.7/site-packages/mercurial/dispatch.py", line 69, in dispatch 19:20:00 INFO - 22:20:00 INFO - ret = _runcatch(req) 19:20:00 INFO - 22:20:00 INFO - File "/tools/python27-mercurial/lib/python2.7/site-packages/mercurial/dispatch.py", line 138, in _runcatch 19:20:00 INFO - 22:20:00 INFO - return _dispatch(req) 19:20:00 INFO - 22:20:00 INFO - File "/tools/python27-mercurial/lib/python2.7/site-packages/mercurial/dispatch.py", line 839, in _dispatch 19:20:00 INFO - 22:20:00 INFO - cmdpats, cmdoptions) 19:20:00 INFO - 22:20:00 INFO - File "/tools/python27-mercurial/lib/python2.7/site-packages/mercurial/dispatch.py", line 600, in runcommand 19:20:00 INFO - 22:20:00 INFO - ret = _runcommand(ui, options, cmd, d) 19:20:00 INFO - 22:20:00 INFO - File "/tools/python27-mercurial/lib/python2.7/site-packages/mercurial/dispatch.py", line 930, in _runcommand 19:20:00 INFO - 22:20:00 INFO - return checkargs() 19:20:00 INFO - 22:20:00 INFO - File "/tools/python27-mercurial/lib/python2.7/site-packages/mercurial/dispatch.py", line 901, in checkargs 19:20:00 INFO - 22:20:00 INFO - return cmdfunc() 19:20:00 INFO - 22:20:00 INFO - File "/tools/python27-mercurial/lib/python2.7/site-packages/mercurial/dispatch.py", line 836, in <lambda> 19:20:00 INFO - 22:20:00 INFO - d = lambda: util.checksignature(func)(ui, *args, **cmdoptions) 19:20:00 INFO - 22:20:00 INFO - File "/tools/python27-mercurial/lib/python2.7/site-packages/mercurial/util.py", line 550, in check 19:20:00 INFO - 22:20:00 INFO - return func(*args, **kwargs) 19:20:00 INFO - 22:20:00 INFO - File "/builds/slave/m-cen-and-api-15-ntly-00000000/scripts/external_tools/robustcheckout.py", line 142, in robustcheckout 19:20:00 INFO - 22:20:00 INFO - sharebase) 19:20:00 INFO - 22:20:00 INFO - File "/builds/slave/m-cen-and-api-15-ntly-00000000/scripts/external_tools/robustcheckout.py", line 213, in _docheckout 19:20:00 INFO - 22:20:00 INFO - makedirs(os.path.dirname(destvfs.base), notindexed=True) 19:20:00 INFO - 22:20:00 INFO - TypeError: ensuredirs() got an unexpected keyword argument 'notindexed' 19:20:00 INFO - 22:20:00 ERROR - Return code: 1 Strangely, a few lines above we're using Mercurial 3.7.3: https://treeherder.mozilla.org/logviewer.html#?job_id=3914824&repo=mozilla-central#L26846 The exception is a bit unfortunate. We could probably teach the robustcheckout extension to sniff the version and fail more gracefully. But that won't solve this bug.
The issue has to do with: 18:05:10 INFO - INFO: installing package(s): autoconf213 mozilla-python27-mercurial yasm ccache zip gcc472_0moz1 gcc473_0moz1 java-1.7.0-openjdk-devel zlib-devel glibc-static openssh-clients mpfr wget glibc.i686 libstdc++.i686 zlib.i686 freetype-2.3.11-6.el6_1.8.x86_64 ant ant-apache-regexp 18:06:02 INFO - INFO: Package mpfr-2.4.1-6.el6.x86_64 already installed and latest version ... 18:06:02 INFO - mozilla-python27-mercurial 18:06:02 INFO - x86_64 3.1.2-1.el6 releng-centos6-x86_64 1.8 M ... 18:06:02 INFO - ================================================================================ 18:06:02 INFO - Install 94 Package(s) 18:06:02 INFO - Total download size: 187 M 18:06:02 INFO - Installed size: 645 M 18:06:02 INFO - Installed: ... 18:06:02 INFO - mozilla-python27-mercurial.x86_64 0:3.1.2-1.el6 I'm not yet sure on *why* that is installing an old mercurial inside mock
To add to Callek's comment, we run the multilocale repacking inside of mock at http://hg.mozilla.org/mozilla-central/file/16663eb3dcfa/testing/mozharness/mozharness/mozilla/building/buildbase.py#l1650 Bug 1229494 installed 3.7.3 at a system level, but it looks like we never updated mock repos so the latest is mozilla-python27-mercurial-3.1.2-1.el6.x86_64.rpm there. dividehex, do you know about the mock repos ? In particular mozilla-releng-mock-centos-us-east-1/releng/public/CentOS/6/x86_64/. How safe or otherwise is it to drop the package we're using for puppet in there ? The packaging looks the same 3.1.2 which is there already, but I'm not sure how we'd regenerate the repodata.
Flags: needinfo?(jwatkins)
(In reply to Nick Thomas [:nthomas] from comment #2) > dividehex, do you know about the mock repos ? In particular > mozilla-releng-mock-centos-us-east-1/releng/public/CentOS/6/x86_64/. Also its pair in mozilla-releng-mock-centos-us-west-2.
(In reply to Nick Thomas [:nthomas] from comment #2) > To add to Callek's comment, we run the multilocale repacking inside of mock > at > http://hg.mozilla.org/mozilla-central/file/16663eb3dcfa/testing/mozharness/ > mozharness/mozilla/building/buildbase.py#l1650 > Bug 1229494 installed 3.7.3 at a system level, but it looks like we never > updated mock repos so the latest is > mozilla-python27-mercurial-3.1.2-1.el6.x86_64.rpm there. > > dividehex, do you know about the mock repos ? In particular > mozilla-releng-mock-centos-us-east-1/releng/public/CentOS/6/x86_64/. How > safe or otherwise is it to drop the package we're using for puppet in there > ? The packaging looks the same 3.1.2 which is there already, but I'm not > sure how we'd regenerate the repodata. I've never updated the mock repos before. In fact, I'm not even sure where the are located. I don't see a problem with dropping in the packages built for puppet. Updating the repo metadata on the puppet server repos is done with 'createrepo --update .' I suspect this would be the same process. https://wiki.mozilla.org/ReleaseEngineering/PuppetAgain/Packages#Ubuntu:_Building_Custom_Packages :dustin might have some more info
Flags: needinfo?(jwatkins) → needinfo?(dustin)
I think there's a mana article about it. You need to drop the files in there and then, iirc, run ./update to copy them to the S3 mirrors. These repos were "forked" from the puppetagain repos a while ago because it seemed like every time we glanced sidelong at the puppetagain repos, the mock builds would fail. So, there be dragons here, but it sounds like this is worth a try.
Flags: needinfo?(dustin)
Blocks: 1273523
In puppet we have repos/yum/custom/mozilla-mercurial/, with 3.7.2+patch and 3.7.3 in it. This should sets up the same for mock, and I'm hoping I can then sync the puppet repo directly to S3. Plus generate new golden images for linux builders.
Attachment #8755619 - Flags: review?(dustin)
Comment on attachment 8755619 [details] [diff] [review] [puppet] Add mozilla-mercurial repo Review of attachment 8755619 [details] [diff] [review]: ----------------------------------------------------------------- This makes great sense. I think you understand this better than I do now :)
Attachment #8755619 - Flags: review?(dustin) → review+
I've copied the puppet repo over to http://mockbuild-repos.pub.build.mozilla.org/custom/mozilla-mercurial/ and synced to S3. Starting new golden image generation now.
The url to the repo needs a /x86_64 suffix on it, which causes errors like this: 17:54:24 INFO - ERROR: Command failed: 17:54:24 INFO - # ['/usr/bin/yum', '--installroot', '/builds/mock_mozilla/mozilla-centos6-x86_64-android/root/', 'install', 'bash', 'bzip2', 'coreutils', 'cpio', 'diffutils', 'fedora-release', 'findutils', 'gawk', 'gmp', 'libstdc++', 'ppl', 'cpp', 'gcc', 'gcc-c++', 'grep', 'gzip', 'info', 'make', 'patch', 'redhat-rpm-config', 'rpm-build', 'sed', 'shadow-utils', 'tar', 'unzip', 'util-linux', 'which', 'xz'] 17:54:24 INFO - https://s3-us-west-2.amazonaws.com/mozilla-releng-mock-centos-us-west-2/custom/mozilla-mercurial/repodata/repomd.xml: [Errno 14] PYCURL ERROR 22 - "The requested URL returned error: 403 Forbidden" 17:54:24 INFO - Trying other mirror. 17:54:24 INFO - Error: Cannot retrieve repository metadata (repomd.xml) for repository: custom-mercurial. Please verify its path and try again 17:54:24 ERROR - Return code: 30 17:54:24 ERROR - 30 not in success codes: [0] 17:54:24 WARNING - setting return code to 3 17:54:24 FATAL - Halting on failure while running ['mock_mozilla', '-r', 'mozilla-centos6-x86_64-android', '--init'] 17:54:24 FATAL - Running post_fatal callback... 17:54:24 FATAL - Exiting 3 I've de-registered the new golden AMIs, and terminated the instances that were running (some try in us-east-1). Will fix that up puppet and try again later.
Fix to repo url landed at: https://hg.mozilla.org/build/puppet/rev/4e6527e4b4bc https://hg.mozilla.org/build/puppet/rev/a4049cbf79a7 I'll let the daily golden image run pick that up at 1:15am Pacific and verify it's OK afterwards.
From a try build with the new golden image (both linux64 and android): 02:47:18 INFO - INFO: installing package(s): autoconf213 python mozilla-python27 zip mozilla-python27-mercurial git ccache perl-Test-Simple perl-Config-General yasm wget mpfr xorg-x11-font* imake gcc45_0moz3 gcc454_0moz1 gcc472_0moz1 gcc473_0moz1 yasm ccache valgrind dbus-x11 glibc-static libstdc++-static gtk2-devel libnotify-devel alsa-lib-devel libcurl-devel wireless-tools-devel libX11-devel libXt-devel mesa-libGL-devel gnome-vfs2-devel GConf2-devel gcc45_0moz3 gcc454_0moz1 gcc472_0moz1 gcc473_0moz1 yasm ccache pulseaudio-libs-devel gstreamer-devel gstreamer-plugins-base-devel freetype-2.3.11-6.el6_1.8.x86_64 freetype-devel-2.3.11-6.el6_1.8.x86_64 02:48:57 INFO - INFO: Package python-2.6.6-29.el6.x86_64 already installed and latest version 02:48:57 INFO - Package mpfr-2.4.1-6.el6.x86_64 already installed and latest version 02:48:57 INFO - ================================================================================ 02:48:57 INFO - Package Arch Version Repository Size 02:48:57 INFO - ================================================================================ 02:48:57 INFO - Installing: 02:48:57 INFO - GConf2-devel x86_64 2.28.0-6.el6 centos6 91 k ... 02:48:57 INFO - mozilla-python27 x86_64 2.7.7-1.el6 releng-centos6-x86_64 11 M 02:48:57 INFO - mozilla-python27-mercurial 02:48:57 INFO - x86_64 3.7.3-1.el6 custom-mercurial 2.2 M So getting hg 3.7.3 in the mock env now. Non-try golden image is still running.
The non-try golden image finished in us-east-1 (ami-7076811d), but we hit the script/AWS bug where the previous golden image was copied to us-west-2 (as ami-9bce30fb). Fixed that up on aws-manager2: >>> from cloudtools.aws.ami import copy_ami, get_ami >>> ami = get_ami('us-east-1', 'bld-linux64') >>> ami.id u'ami-7076811d' >>> new_ami = copy_ami(ami, 'us-west-2') >>> new_ami.id u'ami-c6cf31a6' We can rebuild the android nightly in 3 or so hours, which is long enough for the old instances have recycled themselves.
Assignee: nobody → nthomas
Thank you for tracking this down. Sorry about creating the mess. On the bright side, we're flushing out consumers still using insecure Mercurial versions. So yay for better security.
Breaking Android nightly, tracking.
status-firefox49: --- → affected
tracking-firefox49: --- → blocking
https://treeherder.mozilla.org/#/jobs?repo=mozilla-central&revision=829d3be6ba648b838ee1953fdfa1a477dace752f&filter-job_type_symbol=N shows green now :-) It'd be good to get some spot checking on Italian, Russian, or French in the multi-locale nightly built an hour ago.
I got an update and it seems fine in en-US, please file a followup if there is any l10n bustage.
Status: NEW → RESOLVED
Closed: 9 years ago
Resolution: --- → FIXED
Ioana can your team verify that this is fixed in one of the multi-locale builds mentioned in comment 17? Thanks!
Flags: needinfo?(ioana.chiorean)
Flags: needinfo?(ioana.chiorean) → needinfo?(mihai.ninu)
Yes, Ninu is on it.
Verified as fixed on the 2016-05-24-07-37-14-mozilla-central-android-api-15/ build. This issue was tested on a Xiaomi mi i4 with Android 5.0.2, Nexus 6 with Android 6.0.1 and Nexus 9 tablet with Android 6.0.1. Also the update to the latest Nightly build worked (the one built in on the 25th May, please note that today the Multi build has not yet been built, only the EN-US is present on the ftp servers http://archive.mozilla.org/pub/mobile/nightly/2016/05/2016-05-26-03-02-23-mozilla-central-android-api-15/)
status-firefox49: affected → verified
Flags: needinfo?(mihai.ninu)
You need to log in before you can comment on or make changes to this bug.

Attachment

General

Creator:
Created:
Updated:
Size: