1354216 - cctools job permafail with Artifact "public/github.com/tpoechtrager/cctools-port.tar.gz" not found for task ID undefined

Status: RESOLVED FIXED

(Firefox Build System :: General, defect)

Description

[Sebastian Hengst [:aryx] (needinfo me if it's about an intermittent or backout)]

cctools started running again after almost 2 months and permafails:

https://treeherder.mozilla.org/#/jobs?repo=mozilla-inbound&revision=baade5ccb7c95fb0be9a06e1b8f2a82bf9d19a98&filter-resultStatus=testfailed&filter-resultStatus=busted&filter-resultStatus=exception&filter-resultStatus=retry&filter-resultStatus=usercancel&filter-resultStatus=runnable

> Artifact "public/github.com/tpoechtrager/cctools-port.tar.gz" not found for task ID undefined. This could be caused by the artifact not being created or being marked as expired.

#taskclusters conclusion is that this was already broken before that push.

Comment 1

[Nathan Froyd [:froydnj]]

This is a misleading error; the real problem is before that:

[task 2017-04-06T16:55:04.853717Z] libtool: Version mismatch error.  This is libtool 2.4.6, but the
[task 2017-04-06T16:55:04.853785Z] libtool: definition of this LT_INIT comes from libtool 2.2.6b.
[task 2017-04-06T16:55:04.853820Z] libtool: You should recreate aclocal.m4 with macros from libtool 2.4.6
[task 2017-04-06T16:55:04.853847Z] libtool: and run autoconf again.
[task 2017-04-06T16:55:04.854049Z] Makefile:340: recipe for target 'libstuff_la-allocate.lo' failed

Terrible.  Ehsan, this stuff got added in bug 1331957 and bug 1337233, but the treeherder pages for those landings on inbound and central don't show the cctools job running, which seems quite wrong.  You did build these on infra, right? :)

ni? Dustin too, to see if https://hg.mozilla.org/integration/mozilla-inbound/rev/41e7f431c83d shows something obviously wrong.

Comment 2

[(not currently active) Ted Mielczarek]

I note that a bit up from that in the log there's:
[task 2017-04-06T16:54:54.128353Z] + ./autogen.sh
[task 2017-04-06T16:54:54.130887Z] applying workaround for buggy unistd.h
[task 2017-04-06T16:54:54.133202Z] applying fix for broken unistd.h
[task 2017-04-06T16:54:54.133650Z] + find . -type f '(' -name '*.c' -o -name '*.cpp' -o -name '*.cc' -o -name '*.h' -o -name '*.hpp' ')' -print0
[task 2017-04-06T16:54:54.133707Z] + xargs -0 sed -i 's/#include <unistd.h>/#undef __block\n#include <unistd.h>\n#define __block __attribute__((__blocks__(byref)))/g'
[task 2017-04-06T16:54:54.698139Z] libtoolize: `./config.guess' is newer: use `--force' to overwrite
[task 2017-04-06T16:54:54.705390Z] libtoolize: `./config.sub' is newer: use `--force' to overwrite
[task 2017-04-06T16:54:54.711787Z] libtoolize: `./install-sh' is newer: use `--force' to overwrite
[task 2017-04-06T16:54:54.727231Z] libtoolize: `./ltmain.sh' is newer: use `--force' to overwrite

Comment 3

[Dustin J. Mitchell [:dustin] (he/him)]

I don't see anything obvious.  "What files should cause this to run" is always going to be a difficult question..

Comment 4

[Wes Kocher (:KWierso) (Not reading bugmail; email directly if needed)]

I've hidden these jobs from the default treeherder view. To see them, you'll need to toggle the "excluded jobs" button in Treeherder's header.

Comment 5

[Intermittent Failures Robot]

20 failures in 170 pushes (0.118 failures/push) were associated with this bug yesterday.   

Repository breakdown:
* mozilla-inbound: 20

Platform breakdown:
* toolchains: 20

For more details, see:
https://brasstacks.mozilla.com/orangefactor/?display=Bug&bugid=1354216&startday=2017-04-06&endday=2017-04-06&tree=all

Comment 6

[(no longer active)]

(In reply to Nathan Froyd [:froydnj] from comment #1)
> This is a misleading error; the real problem is before that:
> 
> [task 2017-04-06T16:55:04.853717Z] libtool: Version mismatch error.  This is
> libtool 2.4.6, but the
> [task 2017-04-06T16:55:04.853785Z] libtool: definition of this LT_INIT comes
> from libtool 2.2.6b.
> [task 2017-04-06T16:55:04.853820Z] libtool: You should recreate aclocal.m4
> with macros from libtool 2.4.6
> [task 2017-04-06T16:55:04.853847Z] libtool: and run autoconf again.
> [task 2017-04-06T16:55:04.854049Z] Makefile:340: recipe for target
> 'libstuff_la-allocate.lo' failed
> 
> Terrible.  Ehsan, this stuff got added in bug 1331957 and bug 1337233, but
> the treeherder pages for those landings on inbound and central don't show
> the cctools job running, which seems quite wrong.  You did build these on
> infra, right? :)

I'm usually careful with doing that (and often times don't have a local build environment set up anyway so I can't build these things myself even if I want to!) but to be honest I don't remember what I did exactly in this case any more.  :-(

Also does anyone have a link to a complete log?  And to a TaskClusterId that I can use to run a local docker of the task to try to reproduce the failure locally?

Comment 7

[Sebastian Hengst [:aryx] (needinfo me if it's about an intermittent or backout)]

> Also does anyone have a link to a complete log?  And to a TaskClusterId that
> I can use to run a local docker of the task to try to reproduce the failure
> locally?
The job is now hidden by default on Treeherder, does https://treeherder.mozilla.org/#/jobs?repo=mozilla-inbound&revision=baade5ccb7c95fb0be9a06e1b8f2a82bf9d19a98&filter-resultStatus=testfailed&filter-resultStatus=busted&filter-resultStatus=exception&filter-resultStatus=retry&filter-resultStatus=usercancel&filter-resultStatus=runnable&exclusion_profile=false&selectedJob=89238885 and https://treeherder.mozilla.org/logviewer.html#?job_id=89238885&repo=mozilla-inbound&lineNumber=99 and its artifacts help you?

Comment 8

[Intermittent Failures Robot]

19 failures in 867 pushes (0.022 failures/push) were associated with this bug in the last 7 days.   

Repository breakdown:
* mozilla-inbound: 19

Platform breakdown:
* toolchains: 19

For more details, see:
https://brasstacks.mozilla.com/orangefactor/?display=Bug&bugid=1354216&startday=2017-04-03&endday=2017-04-09&tree=all

Comment 9

[(not currently active) Ted Mielczarek]

https://treeherder.mozilla.org/#/jobs?repo=try&revision=6c53e138b28054b12a45621f9439c0dbb031009f

Comment 10

[(not currently active) Ted Mielczarek]

Attachment: bug 1354216 - fix cctools-port build scripts.

The cctools-port build scripts were pulling and building the master branch
of the cctools-port repo, which means they'd build whatever was there
when they get triggered. I think this was copied from my build-cctools
script which did the same thing, so it's my fault in the end! This patch
pins a revision in the script so we'll build the same thing until we
explicitly update.

I also fixed the scripts to use git instead of tc-vcs, since tc-vcs prints
misleading error messages, and nothing else uses that anymore.

Finally, I removed the build-cctools script, since all the builds are using
cctools-port now so it doesn't serve any useful purpose.

Review commit: https://reviewboard.mozilla.org/r/128610/diff/#index_header
See other reviews: https://reviewboard.mozilla.org/r/128610/

Comment 11

[(not currently active) Ted Mielczarek]

(In reply to Ted Mielczarek [:ted.mielczarek] from comment #9)
> https://treeherder.mozilla.org/#/
> jobs?repo=try&revision=6c53e138b28054b12a45621f9439c0dbb031009f

Both cctools-port builds are green on this push.

Comment 12

[Michael Shal [:mshal]]

Comment on attachment 8856675 [details]
bug 1354216 - fix cctools-port build scripts.

https://reviewboard.mozilla.org/r/128610/#review131040

Comment 13

[Pulsebot]

Pushed by [e-mail]:
https://hg.mozilla.org/integration/autoland/rev/e2bcdd150140
fix cctools-port build scripts. r=mshal

Comment 14

[Carsten Book [:Tomcat]]

https://hg.mozilla.org/mozilla-central/rev/e2bcdd150140

Comment 16

[Carsten Book [:Tomcat]]

also uplifted to 54 in https://hg.mozilla.org/releases/mozilla-beta/rev/7adf8ca53590c43e755d10a4847d6e8a1171ff83 to fix clang errors