1156159 - Investigate hg clone error on AWS 2008 Puppet configured slaves

Status: RESOLVED FIXED

(Infrastructure & Operations :: RelOps: Puppet, task)

Description

[Mark Cornmesser [:markco]]

After getting past win32file error from https://bugzilla.mozilla.org/show_bug.cgi?id=1154755, this comes up: 
 

20:49:08     INFO - Copy/paste: hg --config ui.merge=internal:merge clone https://hg.mozilla.org/build/tools C:/builds/hg-shared\build/tools
20:49:08     INFO - Calling ['hg', '--config', 'ui.merge=internal:merge', 'clone', 'https://hg.mozilla.org/build/tools', 'C:/builds/hg-shared\\build/tools'] with output_timeout 1200
20:49:08     INFO - retry: attempt #5 caught exception: _execute_child() takes exactly 17 arguments (18 given)
20:49:08    FATAL - Automation Error: Can't checkout https://hg.mozilla.org/build/tools!
20:49:08    FATAL - Caught exception: _execute_child() takes exactly 17 arguments (18 given)
20:49:08    FATAL - Caught exception: _execute_child() takes exactly 17 arguments (18 given)
20:49:08    FATAL - Caught exception: _execute_child() takes exactly 17 arguments (18 given)
20:49:08    FATAL - Caught exception: _execute_child() takes exactly 17 arguments (18 given)
20:49:08    FATAL - Caught exception: _execute_child() takes exactly 17 arguments (18 given)
20:49:08    FATAL - Running post_fatal callback...


However, running the command manually it completes: 

C:\mozilla-build\hg>hg --config ui.merge=internal:merge clone https://hg.mozilla
.org/build/tools C:/builds/hg-shared\\build/tools
requesting all changes
adding changesets
adding manifests
adding file changes
added 5650 changesets with 11690 changes to 1597 files
updating to branch default
920 files updated, 0 files merged, 0 files removed, 0 files unresolved

Comment 1

[Dustin J. Mitchell [:dustin] (he/him)]

http://mxr.mozilla.org/build/source/mozharness/mozprocess/processhandler.py#160

That's a code bug - maybe a bogus mozharness commit or something?

Comment 2

[Dustin J. Mitchell [:dustin] (he/him)]

OK, so as far as I can see, _execute_child is defined in mozprocess, which was copied into mozharness a while back

commit 000b8c1d2dd8014606ecd32db85c51927f765334
Author: Aki Sasaki <[e-mail]>
Date:   Wed Jan 29 17:21:47 2014 -0800

    bug 965519 - add mozprocess v0.15 to mozharness. r=jgriffin

I wonder if there's a version mismatch here, where whatever's calling _execute_child (hgtool? it'd be nice if mozharness didn't swallow the traceback) is designed for a different version of mozprocess than what is in use.  I didn't say "installed" because most likely there are several versions of mozprocess installed -- that seems to be de rigeur in our Windows stuff.

jgriffin, any pointers?

Comment 3

[Jonathan Griffin (:jgriffin)]

See https://dxr.mozilla.org/mozilla-central/source/testing/mozbase/mozprocess/mozprocess/processhandler.py#204.  It looks like the caller is expecting the Python >= 2.7.6 version, but is getting the < 2.7.6 version.

Is there a full log for this job?  I might be able to help untangle what's going on.

Comment 4

[Mark Cornmesser [:markco]]

http://dev-master2.bb.releng.use1.mozilla.com:8401/builders/WINNT%205.2%20mozilla-central%20build/builds/359/steps/run_script/logs/stdio

Comment 5

[Dustin J. Mitchell [:dustin] (he/him)]

Ah, I wonder if it's running the "wrong" Python - I think Mozilla-Build still ships Python < 2.7.6, in C:\mozilla-build\python and we install a newer one in c:\mozilla-build\python27, but both are in the PATH.

Given that the <2.7.6 version doesn't work, could we replace it with a simple "Python's too old" failure?

Comment 6

[Jonathan Griffin (:jgriffin)]

I think you may need to merge the latest version of mozprocess into mozharness.  I think what's happening is that current Popen._execute_child requires 18 args, but the "old", Python < 2.7.6 compatible version in mozharness only accepts 17, so the override in mozharness isn't working.

I'm guessing the existing Windows slaves have a Python version that's < 2.7.6 so it isn't running into this problem.

You could try just porting the newer version of _execute_child, if that's easier.

Comment 7

[Dustin J. Mitchell [:dustin] (he/him)]

We're not planning to upgrade Python right now, so I suspect the right answer is to run the same Python version on these systems as we're running on existing Windows slaves.

I suspect this goes back to the bogus PATH entries.

Comment 8

[Dustin J. Mitchell [:dustin] (he/him)]

It looks like this actually has to do with a different python and mozillabuild version on hardware vs. pupppetized windows hosts.  The upgrade was done for NSIS, but led to this issue (and likely others).

I think the best course is to downgrade mozillabuild and deploy the same version with puppet as we currently do on hardware, to limit the number of moving pieces.  Once that's working and in production, the upgrade can come next (at which point, yeah, upgrading mozprocess will be the best fix).

Comment 9

[Mark Cornmesser [:markco]]

This was addressed in BUG 1156153.