1362949 - Build bustage for artifact builds since merge commit c3e5497cff1c: "no built artifacts found"

Status: RESOLVED FIXED

(Firefox Build System :: General, defect)

Description

[Henrik Skupin [:whimboo][⌚️UTC+1]]

I have already noticed this Friday last week and talked to Chris, but then got it accidentally working. So no further action has been taken.

But since morning I cannot build artifact builds on mozilla-central for at least OS X anymore because of:

0:17.27 Attempting to find a pushhead containing c3e5497cff1c995821b1c9320fa71f1ef9a8c30e on mozilla-central.
 0:17.27 Attempting to find a pushhead containing c3e5497cff1c995821b1c9320fa71f1ef9a8c30e on integration/mozilla-inbound.
 0:17.27 Attempting to find a pushhead containing c3e5497cff1c995821b1c9320fa71f1ef9a8c30e on releases/mozilla-beta.
 0:18.26 Retrieving the last 50 pushheads starting with id 90002 on integration/mozilla-inbound
 0:18.26 Retrieving the last 50 pushheads starting with id 31776 on mozilla-central
 0:59.02 Tried 53 pushheads, no built artifacts found.
 0:59.06 make[4]: *** [recurse_artifact] Error 1
 0:59.06 make[3]: *** [artifact] Error 2
 0:59.06 make[2]: *** [default] Error 2
 0:59.06 make[1]: *** [realbuild] Error 2
 0:59.06 make: *** [build] Error 2

This is a merge commit: https://treeherder.mozilla.org/#/jobs?repo=mozilla-central&revision=c3e5497cff1c995821b1c9320fa71f1ef9a8c30e&filter-tier=1&filter-tier=2&filter-tier=3

The merge commit done right before works for me:

https://treeherder.mozilla.org/#/jobs?repo=mozilla-central&revision=22aaf8bad4df8e5f8c976f1521c213cb37e2dff5&filter-tier=1&filter-tier=2&filter-tier=3

But it installs from a remote pushhead from May 5th, which is 5 days old now:

0:03.76 Attempting to find a pushhead containing 22aaf8bad4df8e5f8c976f1521c213cb37e2dff5 on mozilla-central.
 0:03.76 Attempting to find a pushhead containing 22aaf8bad4df8e5f8c976f1521c213cb37e2dff5 on integration/mozilla-inbound.
 0:03.76 Attempting to find a pushhead containing 22aaf8bad4df8e5f8c976f1521c213cb37e2dff5 on releases/mozilla-beta.
 0:04.81 Retrieving the last 50 pushheads starting with id 90002 on integration/mozilla-inbound
 0:04.81 Retrieving the last 50 pushheads starting with id 31775 on mozilla-central
 0:21.56 Installing from remote pushhead 33b92d9c40562dab3d7b602368c75619f1d793f7 on mozilla-central

For me it happens for debug builds while Mark is trying to build opt.

Comment 1

[Chris Manchester (limited bugmail, email directly)]

There are two issues here: one is we stopped producing debug os x buildbot builds, so the artifact code needs to update the artifacts it looks for, and the second, which seems like a real bug, is that we appear to have stopped indexing all the relevant artifacts for buildbot os x opt builds in the taskcluster index, see for instance https://tools.taskcluster.net/task-inspector/#aVkiFaQuRIaVccK5lldcnA/0

Comment 2

[Nick Alexander :nalexander [he/him]]

This looks like fallout from Bug 1350413.  The first build with that ticket has Mac OS X 10.7 opt builds that don't index any artifacts; the tc Tier 2 build that indexed target.dmg is gone:

https://treeherder.mozilla.org/#/jobs?repo=mozilla-central&revision=0b255199db9d6a6f189b89b7906f99155bde3726

Compare to the push before, where there is a tc Tier 2 build indexing target.dmg:

https://treeherder.mozilla.org/#/jobs?repo=mozilla-central&revision=6d4525b5aaa834ea4db8d8eab8087d86e34cf57a&selectedJob=96615068

Comment 3

[Nick Alexander :nalexander [he/him]]

My comment mid-aired with chmanchester's and gives details about his second (real bug).

wcosta, dustin: can you folks dig into the indexing and figure out how to get Mac OS X opt build artifacts back into the Task Cluster index?

Comment 4

[Chris Manchester (limited bugmail, email directly)]

Attachment: Bug 1362949 - Update artifact build code to accept buildbot or taskcluster style artifact names for OS X.

Review commit: https://reviewboard.mozilla.org/r/137192/diff/#index_header
See other reviews: https://reviewboard.mozilla.org/r/137192/

Comment 5

[Chris Manchester (limited bugmail, email directly)]

Comment 4 restores my ability to do debug artifact builds locally.

Comment 6

[Nick Alexander :nalexander [he/him]]

Comment on attachment 8865569 [details]
Bug 1362949 - Update artifact build code to accept buildbot or taskcluster style artifact names for OS X.

https://reviewboard.mozilla.org/r/137192/#review140242

Makes sense.

Comment 7

[Dustin J. Mitchell [:dustin] (he/him)]

Does that patch answer the question?  Anyway, I think Wander has all the state here.

Comment 8

[Pulsebot]

Pushed by [e-mail]:
https://hg.mozilla.org/integration/autoland/rev/b230bcb04a19
Update artifact build code to accept buildbot or taskcluster style artifact names for OS X. r=nalexander

Comment 9

[Wander Lairson Costa]

Is my help still needed here?

Comment 10

[Pulsebot]

Pushed by [e-mail]:
https://hg.mozilla.org/mozilla-central/rev/e0955584782e
Update artifact build code to accept buildbot or taskcluster style artifact names for OS X. r=nalexander a=merge

Comment 11

[Wes Kocher (:KWierso) (Not reading bugmail; email directly if needed)]

https://hg.mozilla.org/mozilla-central/rev/e0955584782e

Comment 12

[Nick Alexander :nalexander [he/him]]

(In reply to Wander Lairson Costa [:wcosta] from comment #9)
> Is my help still needed here?

I think so: chmanchester's patch restores Mac OS X debug builds.  But we don't have any Mac OS X opt builds in the Task Cluster index at this time (that Chris or I can find).  So we need you to either find them for us or get them re-instated in the index, ASAP.

Comment 13

[Chris Manchester (limited bugmail, email directly)]

This is only half fixed. Reopening to avoid confusion.

Comment 14

[Wander Lairson Costa]

Attachment: Fix macosx64-opt index routes. r=dustin

The macosx64-opt build task executes through buildbot-bridge, and there
is a specific task in which artifacts are uploaded.

Comment 15

[Carsten Book [:Tomcat]]

https://hg.mozilla.org/mozilla-central/rev/b230bcb04a19

Comment 16

[Pulsebot]

Pushed by [e-mail]:
https://hg.mozilla.org/integration/mozilla-inbound/rev/9aacb2f027d2
Fix macosx64-opt index routes. r=dustin

Comment 17

[Pulsebot]

Pushed by [e-mail]:
https://hg.mozilla.org/mozilla-central/rev/b9bd73036a92
Fix macosx64-opt index routes. r=dustin a=bustage-fix

Comment 18

[Wes Kocher (:KWierso) (Not reading bugmail; email directly if needed)]

https://hg.mozilla.org/mozilla-central/rev/9aacb2f027d2