1276352 - Fix route/platform names to be consistent between dbg/debug and to conform to existing conventions for platform naming

Status: RESOLVED FIXED

(Firefox Build System :: Task Configuration, task)

Description

[Chris Manchester (limited bugmail, email directly)]

We found a few inconsistencies in https://bugzilla.mozilla.org/show_bug.cgi?id=1276037#c6 that should to be sorted out as we transition more builds.

Comment 1

[Michael Shal [:mshal]]

Attachment: Bug 1276352 - Fix platforms to match buildbot convention;

Review commit: https://reviewboard.mozilla.org/r/57302/diff/#index_header
See other reviews: https://reviewboard.mozilla.org/r/57302/

Comment 2

[Dustin J. Mitchell [:dustin] (he/him)]

https://reviewboard.mozilla.org/r/57300/#review54156

This looks good to me, but Greg may be able to detect some consequences of this change that I don't see.

Comment 3

[Greg Arndt [:garndt]]

Comment on attachment 8759319 [details]
Bug 1276352 - Fix platforms to match buildbot convention;

https://reviewboard.mozilla.org/r/57302/#review54162

If I recall correctly, most fo the build_name and build_type stuff is used for index routes mostly.  Note, that changing these will change the index paths for these tasks in case people are relying on them to be the old naming.

Comment 4

[Chris Manchester (limited bugmail, email directly)]

Mike, I asked about this in the other bug, but it looks like this is changing routes to be consistent with the buildbot names... are we going to start finding TC builds at those routes?

Comment 5

[Michael Shal [:mshal]]

(In reply to Greg Arndt [:garndt] from comment #3)
> If I recall correctly, most fo the build_name and build_type stuff is used
> for index routes mostly.  Note, that changing these will change the index
> paths for these tasks in case people are relying on them to be the old
> naming.

Yeah, the Taskcluster Linux jobs are using new routes that don't line up with the ones that were used by the buildbot jobs (using "linux32" instead of "linux" and "dbg" instead of "debug"). We want to be able to have the Taskcluster builds be able to use those routes, so we don't have to update things that try to pull a "linux-debug" build just because it's now built with Taskcluster. In other words, users of the index shouldn't care whether something was built with Buildbot or Taskcluster - transitioning from one to the other should be seamless.

Please do double-check to make sure I'm actually maintaining consistency here with the Buildbot routes.

Comment 6

[Michael Shal [:mshal]]

(In reply to Chris Manchester (:chmanchester, offline June 3) from comment #4)
> Mike, I asked about this in the other bug, but it looks like this is
> changing routes to be consistent with the buildbot names... are we going to
> start finding TC builds at those routes?

They will once this bug lands and bug 1277666 is fixed. That is what we want, right? Or do you foresee other complications?

Comment 7

[Chris Manchester (limited bugmail, email directly)]

Artifact builds look for artifacts with names with certain names, so those might break (in other words, we'd need a patch like the one in bug 1276037 wherever this happens).

Comment 8

[Henrik Skupin [:whimboo][⌚️UTC+1]]

So doing the changes as listed in the patch would indeed give us the same routes BUT it will not allow us anymore to retrieve any single tc task from the index. This is because of bug 1274311 and buildbot tasks always overwriting the routes of already finished tc tasks.

Comment 9

[Michael Shal [:mshal]]

(In reply to Henrik Skupin (:whimboo) from comment #8)
> So doing the changes as listed in the patch would indeed give us the same
> routes BUT it will not allow us anymore to retrieve any single tc task from
> the index. This is because of bug 1274311 and buildbot tasks always
> overwriting the routes of already finished tc tasks.

Can you clarify how you're using the index? Most users of it do not want to distinguish between buildbot and taskcluster, but rather just find (for example) a linux debug build for a specific revision or date. The route to get a linux debug build should not change as a build is moved from buildbot to taskcluster.

Comment 10

[Henrik Skupin [:whimboo][⌚️UTC+1]]

As long as we still run buildbot tasks for the same kind of job, and both tc and buildbot indexing the same routes, only one of both will win. And as it looks like it is buildbot.

As an example see:

https://tools.taskcluster.net/index/artifacts/#gecko.v2.mozilla-central.revision.5f95858f8ddf21ea2271a12810332efd09eff138.firefox/gecko.v2.mozilla-central.revision.5f95858f8ddf21ea2271a12810332efd09eff138.firefox.linux64-opt

Comment 11

[Michael Shal [:mshal]]

They are supposed to use the same route, so that when Taskcluster is promoted to tier 1, we can turn off the equivalent buildbot build and all clients of the index will continue to get the builds they need - they just happen to be built by Taskcluster instead of buildbot.

Is there a client of the index that only wants Taskcluster builds, even if they are tier 2? Or was something in TC promoted to tier 1 and no-one turned off the buildbot build?

Comment 12

[Henrik Skupin [:whimboo][⌚️UTC+1]]

I see. So this is also a Tier level thing, and the higher level has precedence. I think that clears it up.

So for mozmill-ci I have to use the Taskcluster client to find build jobs as done by Taskcluster itself. It's necessary to retrieve the docker image taskid, which means buildbot won't work. Given the behavior above I cannot rely on any index but only on those where the tc build has tier-1, which at the moment is debug only. Given the name mismatch as covered by this bug I also have to use 'dbg' and not 'debug'. 

Anyway, I think that the above is fine and I can update our code for mozilla-central. For mozilla-aurora I would still have to use 'dbg' until the next version bump and tree merge.

Comment 13

[Pulsebot]

Pushed by [e-mail]:
https://hg.mozilla.org/integration/mozilla-inbound/rev/f9d32d5fc19a
Fix platforms to match buildbot convention; r=dustin, garndt

Comment 14

[Sebastian Hengst [:aryx] (needinfo me if it's about an intermittent or backout)]

Backed out for build bustage:

https://hg.mozilla.org/integration/mozilla-inbound/rev/bec66d66db94

Push with failure: https://treeherder.mozilla.org/#/jobs?repo=mozilla-inbound&revision=f9d32d5fc19a8cd5e408f7ff05283ff0c837061c
Failure log: https://treeherder.mozilla.org/logviewer.html#?job_id=31003958&repo=mozilla-inbound

/home/worker/workspace/build/src/MacOSX10.7.sdk/usr/include/c++/v1/atomic:1353:51: error: too few arguments to function call, expected 4, have 3
...
21:55:08     INFO -  /home/worker/workspace/build/src/obj-firefox/dist/include/mozilla/ChaosMode.h:55:9: note: in instantiation of member function 'mozilla::detail::AtomicBaseIncDec<unsigned int, mozilla::MemoryOrdering::SequentiallyConsistent>::operator unsigned int' requested here
21:55:08     INFO -      if (detail::gChaosModeCounter > 0) {
21:55:08     INFO -          ^
21:55:08     INFO -  5 errors generated.
21:55:08     INFO -  gmake[5]: *** [Unified_cpp_mfbt0.o] Error 1
21:55:08     INFO -  gmake[5]: Leaving directory `/home/worker/workspace/build/src/obj-firefox/mfbt'

Comment 15

[Michael Shal [:mshal]]

It looks like there is a rule in treeherder make 'macosx64-dbg' tier-2 and ignored, but by turning this into 'macosx64-debug', we miss that rule. Taskcluster should be marking these builds tier-2, and we probably need a new treeherder rule to ignore 'macosx64-debug' as well.

Comment 16

[Pulsebot]

Pushed by [e-mail]:
https://hg.mozilla.org/integration/mozilla-inbound/rev/99a891cdb4fe
Fix platforms to match buildbot convention; r=dustin, garndt

Comment 17

[Carsten Book [:Tomcat]]

https://hg.mozilla.org/mozilla-central/rev/99a891cdb4fe

Comment 18

[Henrik Skupin [:whimboo][⌚️UTC+1]]

Michael, I see a similar thing for pgo opt builds where we have two entries right now:

https://tools.taskcluster.net/index/artifacts/#gecko.v2.mozilla-central.revision.39dffbba764210b25bfc1e749b4f16db77fa0d46.firefox/gecko.v2.mozilla-central.revision.39dffbba764210b25bfc1e749b4f16db77fa0d46.firefox.linux64-pgo-opt

https://tools.taskcluster.net/index/artifacts/#gecko.v2.mozilla-central.revision.39dffbba764210b25bfc1e749b4f16db77fa0d46.firefox/gecko.v2.mozilla-central.revision.39dffbba764210b25bfc1e749b4f16db77fa0d46.firefox.linux64-pgo

I assume this needs a similar fix?

Comment 19

[Michael Shal [:mshal]]

(In reply to Henrik Skupin (:whimboo) from comment #18)
> I assume this needs a similar fix?

Yes, thanks for catching that! I filed bug 1284602 and am testing the fix now.