Closed Bug 1330837 Opened 5 years ago Closed 5 years ago

Update tools tests to use mercurial 4.1

Categories

(Infrastructure & Operations Graveyard :: CIDuty, task)

task
Not set
normal

Tracking

(firefox55 fixed)

RESOLVED FIXED
Tracking Status
firefox55 --- fixed

People

(Reporter: nthomas, Assigned: gps)

References

Details

Attachments

(3 files)

We upgraded from Mercurial 3.7.3 to 3.9.1 in Sep/Oct 2016 (bug 1298976) but the tools tests still use 3.7.3. See tox.ini in the base of tools for where we specify 3.7.3.

We'll have to fix this:

23) ERROR: testApplyAndPushRebaseFails (mozilla_buildtools.test.test_util_hg.TestHg)

   Traceback (most recent call last):
    lib/python/mozilla_buildtools/test/test_util_hg.py line 660 in testApplyAndPushRebaseFails
      lambda r, a: c(r, a, self.repodir), max_attempts=3)
    lib/python/util/hg.py line 564 in apply_and_push
      raise HgUtilError("Failed to push")
   HgUtilError: Failed to push
   -------------------- >> begin captured logging << --------------------
   util.commands: INFO: command: START
   util.commands: INFO: command: /Users/nthomas/source/hgmo/build/tools/lib/python/mozilla_buildtools/test/init_hgrepo.sh /var/folders/gr/f_63bw8j6jx1kbs37gfsvhf00000gn/T/tmplsLVK5/repo
   util.commands: INFO: command: cwd: /private/var/folders/gr/f_63bw8j6jx1kbs37gfsvhf00000gn/T/tmplsLVK5
   util.commands: INFO: command: output:
   util.commands: INFO: command: END (1.00s elapsed)

   util.commands: INFO: command: START
   util.commands: INFO: command: hg log -R /var/folders/gr/f_63bw8j6jx1kbs37gfsvhf00000gn/T/tmplsLVK5/repo --template {node|short}

   util.commands: INFO: command: cwd: /private/var/folders/gr/f_63bw8j6jx1kbs37gfsvhf00000gn/T/tmplsLVK5
   util.commands: DEBUG: Process returned 0
   util.commands: INFO: command: END (0.25s elapsed)

   util.commands: INFO: command: output:
   util.commands: INFO: d9647854da5c
   708108b19859
   f9129973d36c
   f99e43af0df3

   redo: DEBUG: attempt 1/2
   util.commands: INFO: command: START
   util.commands: INFO: command: hg clone --traceback /var/folders/gr/f_63bw8j6jx1kbs37gfsvhf00000gn/T/tmplsLVK5/repo /var/folders/gr/f_63bw8j6jx1kbs37gfsvhf00000gn/T/tmplsLVK5/wc
   util.commands: INFO: command: cwd: /private/var/folders/gr/f_63bw8j6jx1kbs37gfsvhf00000gn/T/tmplsLVK5
   util.commands: INFO: command: env: {'HGPLAIN': '1'}
   util.commands: DEBUG: Process returned 0
   util.commands: INFO: command: END (0.12s elapsed)

   util.commands: INFO: command: output:
   util.commands: INFO: updating to branch default
   2 files updated, 0 files merged, 0 files removed, 0 files unresolved

   util.commands: INFO: command: START
   util.commands: INFO: command: hg branch
   util.commands: INFO: command: cwd: /var/folders/gr/f_63bw8j6jx1kbs37gfsvhf00000gn/T/tmplsLVK5/wc
   util.commands: INFO: command: env: {'HGPLAIN': '1'}
   util.commands: DEBUG: Process returned 0
   util.commands: INFO: command: END (0.09s elapsed)

   util.commands: INFO: command: output:
   util.commands: INFO: default

   util.commands: INFO: command: START
   util.commands: INFO: command: hg update -C
   util.commands: INFO: command: cwd: /var/folders/gr/f_63bw8j6jx1kbs37gfsvhf00000gn/T/tmplsLVK5/wc
   util.commands: INFO: command: output:
   util.commands: INFO: command: END (0.12s elapsed)

   util.commands: INFO: command: START
   util.commands: INFO: command: hg parent --template {node|short}
   util.commands: INFO: command: cwd: /var/folders/gr/f_63bw8j6jx1kbs37gfsvhf00000gn/T/tmplsLVK5/wc
   util.commands: INFO: command: env: {'HGPLAIN': '1'}
   util.commands: DEBUG: Process returned 0
   util.commands: INFO: command: END (0.10s elapsed)

   util.commands: INFO: command: output:
   util.commands: INFO: d9647854da5c
   util.commands: INFO: command: START
   util.commands: INFO: command: hg branch
   util.commands: INFO: command: cwd: /var/folders/gr/f_63bw8j6jx1kbs37gfsvhf00000gn/T/tmplsLVK5/wc
   util.commands: INFO: command: env: {'HGPLAIN': '1'}
   util.commands: DEBUG: Process returned 0
   util.commands: INFO: command: END (0.09s elapsed)

   util.commands: INFO: command: output:
   util.commands: INFO: default

   util.commands: INFO: command: START
   util.commands: INFO: command: hg tag -f TEST
   util.commands: INFO: command: cwd: /var/folders/gr/f_63bw8j6jx1kbs37gfsvhf00000gn/T/tmplsLVK5/wc
   util.commands: INFO: command: output:
   util.commands: INFO: command: END (0.14s elapsed)

   util.commands: INFO: command: START
   util.commands: INFO: command: hg tag -f CONFLICTING_TAG
   util.commands: INFO: command: cwd: /var/folders/gr/f_63bw8j6jx1kbs37gfsvhf00000gn/T/tmplsLVK5/repo
   util.commands: INFO: command: output:
   util.commands: INFO: command: END (0.14s elapsed)

   util.commands: INFO: command: START
   util.commands: INFO: command: hg -q out --template "{node} {branches}
   " /var/folders/gr/f_63bw8j6jx1kbs37gfsvhf00000gn/T/tmplsLVK5/repo
   util.commands: INFO: command: cwd: /var/folders/gr/f_63bw8j6jx1kbs37gfsvhf00000gn/T/tmplsLVK5/wc
   util.commands: INFO: command: env: {'HGPLAIN': '1'}
   util.commands: DEBUG: Process returned 0
   util.commands: INFO: command: END (0.10s elapsed)

   util.commands: INFO: command: output:
   util.commands: INFO: e3d64a40f316ba041c7ed41e09d0b5da4813704e

   util.commands: INFO: command: START
   util.commands: INFO: command: hg push --traceback --new-branch /var/folders/gr/f_63bw8j6jx1kbs37gfsvhf00000gn/T/tmplsLVK5/repo
   util.commands: INFO: command: cwd: /var/folders/gr/f_63bw8j6jx1kbs37gfsvhf00000gn/T/tmplsLVK5/wc
   util.commands: INFO: command: output:
   util.commands: INFO: command: ERROR
   Traceback (most recent call last):
     File "/Users/nthomas/source/hgmo/build/tools/lib/python/util/commands.py", line 52, in run_cmd
       return subprocess.check_call(cmd, **kwargs)
     File "/usr/local/Cellar/python/2.7.11/Frameworks/Python.framework/Versions/2.7/lib/python2.7/subprocess.py", line 540, in check_call
       raise CalledProcessError(retcode, cmd)
   CalledProcessError: Command '['hg', 'push', '--traceback', '--new-branch', '/var/folders/gr/f_63bw8j6jx1kbs37gfsvhf00000gn/T/tmplsLVK5/repo']' returned non-zero exit status 255
   util.commands: INFO: command: END (0.13s elapsed)

   util.hg: DEBUG: Hit error when trying to push: Command '['hg', 'push', '--traceback', '--new-branch', '/var/folders/gr/f_63bw8j6jx1kbs37gfsvhf00000gn/T/tmplsLVK5/repo']' returned non-zero exit status 255
Component: General Automation → Buildduty
QA Contact: catlee → bugspam.Callek
Assignee: nobody → aselagea
Alin you asked about this in our stand up today but I don't see any messages about the errors your ran into.  Could you add them to help debug?
What I'd really like to know here is how to run these tests and how are they used in production.

Here are the steps that I followed:
    1. Cloned /tools repo
    2. Created a new py27 environment inside that repo and activated that env
    3. Ran “python setup.py develop”
    4. Ran "pip install tox"
    5. Ran “tox”

I tried 3 scenarios to run these tests:
    - on my local machine (using Cygwin): I managed to follow all the steps above, but I received something like: “360 tests run in 649.7 seconds. 3 FAILED, 6 errors (351 tests passed)” (full log attached). Nick's test failure is not among the ones that failed in my case
    - on a CentOS local VM: it doesn't manage to install one of the dependencies.

    gevent/libevent.h:9:19: fatal error: event.h: No such file or directory
     #include "event.h"
                       ^
    compilation terminated.
    error: command 'gcc' failed with exit status 1

    - on a CentOS loaner ("dev-linux64-ec2-aselagea"): I had to manually install/update some packages before being able to install tox and run it. And even so, it will stuck installing the first dependency:

Collecting nose==1.3.0
  Could not find a version that satisfies the requirement nose==1.3.0 (from versions: 1.3.7)
No matching distribution found for nose==1.3.0
 
I feel like I'm missing something here though.
I'm on a mac.  I ran this to fix the error above

brew install libevent

but they ran into another error with another lib that apparently isn't supported on mac.  So I would suggest checking out a loaner, running the steps above, and then adding the new libraries as required by the new python version
4.1 is now deployed in many parts of automation. So when we bump 3.7.3 to something, it should be 4.1.
Summary: Update tools tests to use mercurial 3.9.1 → Update tools tests to use mercurial 4.1
This bug has been idle for 3 months and I got bit by this today. So I'm stealing it.
Assignee: aselagea → gps
Status: NEW → ASSIGNED
Comment on attachment 8857198 [details]
Bug 1330837 - Remove functionality to apply and push;

https://reviewboard.mozilla.org/r/129124/#review131634

It looks like we don't use the mozharness apply_and_push anywhere.
Attachment #8857198 - Flags: review?(aki) → review+
Comment on attachment 8857199 [details]
Bug 1330837 - Use Mercurial 4.1 in mozharness tests;

https://reviewboard.mozilla.org/r/129126/#review131640

I was wondering if we needed to upload 4.1.2 to internal pypi, but it looks like 3.7.3 isn't there, so this should be status quo.
It looks like we can install 3.7.3 and 3.9.1 via puppet RPMs on releng infra.
Attachment #8857199 - Flags: review?(aki) → review+
I need to file bugs to get buildbot infra on 4.1. So far, I haven't cared enough. And I'm still not convinced it is worth caring.
Pushed by gszorc@mozilla.com:
https://hg.mozilla.org/integration/autoland/rev/a6a2df0ee343
Remove functionality to apply and push; r=aki
https://hg.mozilla.org/integration/autoland/rev/f7b367022241
Use Mercurial 4.1 in mozharness tests; r=aki
scripts/release/tag-release.py from build/tools references apply_and_push(). So I'm hesitant to remove it.
(In reply to Gregory Szorc [:gps] from comment #14)
> scripts/release/tag-release.py from build/tools references apply_and_push().
> So I'm hesitant to remove it.

Understood.  However, the summary of this bug is to fix the tools tests, and we've fixed the mozharness tests; should we reopen or file new to continue to track the tools tests?
(In reply to Aki Sasaki [:aki] from comment #16)
> Understood.  However, the summary of this bug is to fix the tools tests, and
> we've fixed the mozharness tests; should we reopen or file new to continue
> to track the tools tests?

Bleh. I didn't notice. I forgot we had 2 things that were essentially the same.

Yes, we should open a new bug to track getting tools upgraded. But this will be more difficult since I believe some things running with tools are still clinging to an old Mercurial version :(
Blocks: 1356058
See Also: → 1356058
Product: Release Engineering → Infrastructure & Operations
Product: Infrastructure & Operations → Infrastructure & Operations Graveyard
You need to log in before you can comment on or make changes to this bug.