1154669 - Taskcluster B2g Build fails with bustage from b2g-inbound in other unrelated trees

Status: RESOLVED FIXED

(Taskcluster :: General, defect, P1)

Description

[Carsten Book [:Tomcat]]

-> We had bustage in b2g-i like https://treeherder.mozilla.org/logviewer.html#?job_id=1724796&repo=b2g-inbound (Bug 1153796) and that got backed out
-> while the b2g-inbound builds are now fine, suddenly builds on fx-team and mozilla-inbound fails with:

https://treeherder.mozilla.org/logviewer.html#?job_id=8872723&repo=mozilla-inbound

while there was no merge or something that could explain that cross tree failure.

03:49 < tzimmermann> it almost seems like the moz.build was merged into fx-team, but not the actual source files
03:54 < tzimmermann> fx-team and m-i seem to execute this line: https://hg.mozilla.org/integration/b2g-inbound/file/bccb220e6859/dom/bluetooth/moz.build#l14

Comment 1

[Carsten Book [:Tomcat]]

Pete: could you take a look into this

Comment 2

[Legacy TBPL/Treeherder Robot]

log: https://treeherder.mozilla.org/logviewer.html#?repo=mozilla-inbound&job_id=8872723
repository: mozilla-inbound
start_time: 2015-04-15T03:33:12
who: tomcat[at]mozilla[dot]com
machine: unknown
revision: fa98fe5c4ca2

Errors received:
The directory '/home/worker/.cache/pip/log' or its parent directory is not owned by the current user and the debug log has been disabled. Please check the permissions and owner of that directory. If executing pip with sudo, you may want the -H flag.
The directory '/home/worker/.cache/pip/http' or its parent directory is not owned by the current user and the cache has been disabled. Please check the permissions and owner of that directory. If executing pip with sudo, you may want the -H flag.
You are using pip version 6.0.8, however version 6.1.1 is available.
You should consider upgrading via the 'pip install --upgrade pip' command.
Return code: 0
IOError: [Errno 2] No such file or directory: '/home/worker/workspace/gecko/testing/taskcluster/scripts/builder/oauth.txt'
make[6]: *** No rule to make target `../../../../gecko/dom/bluetooth/BluetoothInterface.cpp', needed by `BluetoothInterface.o'.  Stop.
make[6]: *** Waiting for unfinished jobs....
make[5]: *** [dom/bluetooth/target] Error 2
make[5]: *** Waiting for unfinished jobs....
make[4]: *** [compile] Error 2
make[3]: *** [default] Error 2
make[2]: *** [realbuild] Error 2
make[1]: *** [build] Error 2
make: *** [out/target/product/generic/obj/DATA/gecko_intermediates/gecko] Error 2
Return code: 2
failed to build
Running post_fatal callback...
Exiting 2

Comment 3

[Legacy TBPL/Treeherder Robot]

log: https://treeherder.mozilla.org/logviewer.html#?repo=mozilla-inbound&job_id=8873467
repository: mozilla-inbound
start_time: 2015-04-15T03:49:42
who: tomcat[at]mozilla[dot]com
machine: unknown
revision: 84d45d5357f4

Errors received:
The directory '/home/worker/.cache/pip/log' or its parent directory is not owned by the current user and the debug log has been disabled. Please check the permissions and owner of that directory. If executing pip with sudo, you may want the -H flag.
The directory '/home/worker/.cache/pip/http' or its parent directory is not owned by the current user and the cache has been disabled. Please check the permissions and owner of that directory. If executing pip with sudo, you may want the -H flag.
You are using pip version 6.0.8, however version 6.1.1 is available.
You should consider upgrading via the 'pip install --upgrade pip' command.
Return code: 0
IOError: [Errno 2] No such file or directory: '/home/worker/workspace/gecko/testing/taskcluster/scripts/builder/oauth.txt'
make[6]: *** No rule to make target `../../../../gecko/dom/bluetooth/BluetoothInterface.cpp', needed by `BluetoothInterface.o'.  Stop.
make[6]: *** Waiting for unfinished jobs....
make[5]: *** [dom/bluetooth/target] Error 2
make[5]: *** Waiting for unfinished jobs....
make[4]: *** [compile] Error 2
make[3]: *** [default] Error 2
make[2]: *** [realbuild] Error 2
make[1]: *** [build] Error 2
make: *** [out/target/product/generic/obj/DATA/gecko_intermediates/gecko] Error 2
Return code: 2
failed to build
Running post_fatal callback...
Exiting 2

Comment 4

[Legacy TBPL/Treeherder Robot]

log: https://treeherder.mozilla.org/logviewer.html#?repo=fx-team&job_id=2686685
repository: fx-team
start_time: 2015-04-15T04:05:52
who: tomcat[at]mozilla[dot]com
machine: unknown
revision: 9d319ca0f937

Errors received:
The directory '/home/worker/.cache/pip/log' or its parent directory is not owned by the current user and the debug log has been disabled. Please check the permissions and owner of that directory. If executing pip with sudo, you may want the -H flag.
The directory '/home/worker/.cache/pip/http' or its parent directory is not owned by the current user and the cache has been disabled. Please check the permissions and owner of that directory. If executing pip with sudo, you may want the -H flag.
You are using pip version 6.0.8, however version 6.1.1 is available.
You should consider upgrading via the 'pip install --upgrade pip' command.
Return code: 0
IOError: [Errno 2] No such file or directory: '/home/worker/workspace/gecko/testing/taskcluster/scripts/builder/oauth.txt'
make[6]: *** No rule to make target `../../../../gecko/dom/bluetooth/BluetoothInterface.cpp', needed by `BluetoothInterface.o'.  Stop.
make[5]: *** [dom/bluetooth/target] Error 2
make[5]: *** Waiting for unfinished jobs....
make[4]: *** [compile] Error 2
make[3]: *** [default] Error 2
make[2]: *** [realbuild] Error 2
make[1]: *** [build] Error 2
make: *** [out/target/product/generic/obj/DATA/gecko_intermediates/gecko] Error 2
Return code: 2
failed to build
Running post_fatal callback...
Exiting 2

Comment 5

[Greg Arndt [:garndt]]

Looking at one particular node,

This b2g-inbound build completed successfully:
http://docs.taskcluster.net/tools/task-inspector/#SUQsMe_RT_2Rl7gH5-1G4w/1

Then the same cache was used for this m-c build that failed:
http://docs.taskcluster.net/tools/task-inspector/#dhQffUFhTOmcx0TtWBAvsQ/0

After that this fx-team build failed using that cache:
http://docs.taskcluster.net/tools/task-inspector/#TFt50lxORRqvtF75LJc0gQ/0

The failed jobs above, and the failed b2g-inbound that triggered a backout have this same line (which seems to agree with the command about moz.build above):
 ERROR -  make[6]: *** No rule to make target `../../../../gecko/dom/bluetooth/bluetooth1/BluetoothInterface.cpp', needed by `BluetoothInterface.o'.  Stop.


 Both failed jobs for m-c and fx-team did not do a clobber (there is a line for skipping clobbering).

Comment 6

[Greg Arndt [:garndt]]

(In reply to Greg Arndt [:garndt] from comment #5)
> Then the same cache was used for this m-c build that failed:
> http://docs.taskcluster.net/tools/task-inspector/#dhQffUFhTOmcx0TtWBAvsQ/0

Correction, that build is for a try push.

Comment 7

[Legacy TBPL/Treeherder Robot]

log: https://treeherder.mozilla.org/logviewer.html#?repo=fx-team&job_id=2687309
repository: fx-team
start_time: 2015-04-15T04:54:56
who: tomcat[at]mozilla[dot]com
machine: unknown
revision: 75cc30c9a401

Errors received:
The directory '/home/worker/.cache/pip/log' or its parent directory is not owned by the current user and the debug log has been disabled. Please check the permissions and owner of that directory. If executing pip with sudo, you may want the -H flag.
The directory '/home/worker/.cache/pip/http' or its parent directory is not owned by the current user and the cache has been disabled. Please check the permissions and owner of that directory. If executing pip with sudo, you may want the -H flag.
You are using pip version 6.0.8, however version 6.1.1 is available.
You should consider upgrading via the 'pip install --upgrade pip' command.
Return code: 0
IOError: [Errno 2] No such file or directory: '/home/worker/workspace/gecko/testing/taskcluster/scripts/builder/oauth.txt'
make[6]: *** No rule to make target `../../../../gecko/dom/bluetooth/BluetoothInterface.cpp', needed by `BluetoothInterface.o'.  Stop.
make[5]: *** [dom/bluetooth/target] Error 2
make[5]: *** Waiting for unfinished jobs....
make[4]: *** [compile] Error 2
make[3]: *** [default] Error 2
make[2]: *** [realbuild] Error 2
make[1]: *** [build] Error 2
make: *** [out/target/product/generic/obj/DATA/gecko_intermediates/gecko] Error 2
make: *** Waiting for unfinished jobs....
Return code: 2
failed to build
Running post_fatal callback...
Exiting 2

Comment 8

[Ryan VanderMeulen [:RyanVM]]

This is a critical issue with shared objdirs that needs fixing (if it hasn't been already).

Comment 9

[Ryan VanderMeulen [:RyanVM]]

Hit this again today with a push to inbound that broke fx-team.

The push:
https://treeherder.mozilla.org/#/jobs?repo=mozilla-inbound&revision=319ea1367089

Was a needs-clobber push. The fx-team bustage:
https://tools.taskcluster.net/task-inspector/#q8YOH37xRPeuXlJVfouDRQ/0

Bug 1132574 isn't assigned to anybody. Should I be worried?

Comment 10

[Selena Deckelmann :selenamarie :selena]

(In reply to Ryan VanderMeulen [:RyanVM UTC-4] from comment #9)

> Bug 1132574 isn't assigned to anybody. Should I be worried?

No :) 

:garndt is working on it.

Comment 11

[Selena Deckelmann :selenamarie :selena]

Resolving fixed based on objdir work and automatic clobber on try being implemented.