1371300 - Build bustage all platforms on 2017-06-08

Status: RESOLVED FIXED

(Thunderbird :: Build Config, defect)

Description

[Jorg K (CEST = GMT+2)]

All platforms fail, but the errors are different:

Linux 64:
make[5]: *** [mozmill-virtualenv] Error 1

../../../config/nsinstall: cannot access /builds/slave/tb-c-cen-l64-00000000000000000/build/mozilla/python/virtualenv: No such file or directory
make[5]: *** [mozmill-virtualenv] Error 1

Mac:
ImportError: No module named which
make[2]: *** [backend.RecursiveMakeBackend] Error 1
make[1]: *** [build] Error 2
make: *** [build] Error 2
program finished with exit code 2

Windows:
IOError: [Errno 2] No such file or directory: u'c:\\builds\\moz2_slave\\tb-c-cen-w32-00000000000000000\\build\\mozilla\\python\\virtualenv' 

Maybe a buildbot change. M-C regression interval:

Last good: 7efda263a842
First bad: 6491fb29e7fc

Range:
https://hg.mozilla.org/mozilla-central/pushloghtml?fromchange=7efda263a842&tochange=6491fb29e7fc

The only build related bug in the range is bug 1356952. But that only changed files under /taskcluster. Oh, there's also bug 1370421.

Do putting NI for some people who might know something about this.

Off-topic: The TB Council is currently conducting job interviews for a build/release engineer. So hopefully pretty soon we'll know who to call if something like this happens.

Comment 1

[Jorg K (CEST = GMT+2)]

Richard reports a failing local build:

 0:05.98 ImportError: No module named which
 0:05.98 z:/Mozilla/comm-central/mozilla/build/rebuild-backend.mk:24:
recipe for target 'backend.RecursiveMakeBackend' failed
 0:05.98 mozmake.EXE[1]: *** [backend.RecursiveMakeBackend] Error 1
 0:05.99 client.mk:397: recipe for target 'build' failed
 0:05.99 mozmake.EXE: *** [build] Error 2
 0:06.01 266 compiler warnings present.

Comment 2

[Jorg K (CEST = GMT+2)]

Richard reckons it's from bug 1346025.

Comment 3

[Jorg K (CEST = GMT+2)]

Hmm, I did a clobber on treeherder C-C and retriggered a job which has been running for ten minutes now where the failed job took two minutes. So perhaps a false alarm?

Comment 4

[Gregory Szorc [:gps]]

Yes, a clobber may be needed. Also, audit your code for references to "sys.path" and "python/" and adjust moved paths accordingly.

Comment 5

[Jorg K (CEST = GMT+2)]

We have a few usages of sys.path, but perhaps more importantly:
mail/test/mozmill/Makefile.in
22 $(INSTALL) $(topsrcdir)/mozilla/python/virtualenv $(_DEST_DIR)/resources/

That looks like the error from comment #0:
cannot access /builds/slave/tb-c-cen-l64-00000000000000000/build/mozilla/python/virtualenv

So that's now in /third_party/python.

Comment 6

[Jorg K (CEST = GMT+2)]

Let's see where we get with this:
https://hg.mozilla.org/comm-central/rev/998749e6ed4e8c8a70b406fa421cf64e98f0977a

Comment 7

[Jorg K (CEST = GMT+2)]

That did the trick. |hg qfinish -a| ate my patch, so no attachment here ;-)

One liner:
-	$(INSTALL) $(topsrcdir)/mozilla/python/virtualenv $(_DEST_DIR)/resources/
+	$(INSTALL) $(topsrcdir)/mozilla/third_party/python/virtualenv $(_DEST_DIR)/resources/

Comment 9

[ISHIKAWA, Chiaki]

This patch is in C-C tree, correct?
I updated local C-C tree by |make -f client.mk checkout| but it did not get fetched. Oh well.

TIA

Comment 10

[Jorg K (CEST = GMT+2)]

Then you need to clean your local repository: Do "hg out" and "hg strip" and "hg pull" and "hg up -C default".

Comment 11

[ISHIKAWA, Chiaki]

It seems that there is a slight delay between the patch being submitted and the (outward) server from which I pick up
the recent C-C tree.
I did the following as you suggested (I did not understand what the commands do, but after seeing the messages I think I do now.)

--- begin quote ---
ishikawa@ip030:/NREF-COMM-CENTRAL/comm-central$ hg out
comparing with http://hg.mozilla.org/comm-central
real URL is https://hg.mozilla.org/comm-central
searching for changes
no changes found
ishikawa@ip030:/NREF-COMM-CENTRAL/comm-central$ hg strip
abort: empty revision set
ishikawa@ip030:/NREF-COMM-CENTRAL/comm-central$ hg pull
pulling from http://hg.mozilla.org/comm-central
real URL is https://hg.mozilla.org/comm-central
searching for changes
no changes found
ishikawa@ip030:/NREF-COMM-CENTRAL/comm-central$ hg up -C default
0 files updated, 0 files merged, 0 files removed, 0 files unresolved
ishikawa@ip030:/NREF-COMM-CENTRAL/comm-central$ 

--- end quote ---

Oh well, maybe it would work now ...