Closed Bug 1276037 Opened 4 years ago Closed 4 years ago

Can't build debug on Linux64 because the artifact names changed when we switched to taskcluster

Categories

(Firefox Build System :: General, defect)

defect
Not set

Tracking

(firefox49 fixed)

RESOLVED FIXED
mozilla49
Tracking Status
firefox49 --- fixed

People

(Reporter: Gijs, Assigned: chmanchester)

References

Details

Attachments

(3 files)

Looking at:

https://tools.taskcluster.net/index/artifacts/#gecko.v2.mozilla-central.latest.firefox

it seems we now have items for:

linux64-dbg

and

linux64-debug

and the later has not been updated for 2 weeks now, which coincides nicely (in my memory, at least) with when we switched to taskcluster for those builds.

This means that if you use a linux64 debug artifact build, you're now SOL, because this happens:

 0:15.87 Attempting to find a pushhead containing b0096c5c727749ad3e79cbdf20d2e96bd179c213 on mozilla-central.
 0:15.87 Attempting to find a pushhead containing b0096c5c727749ad3e79cbdf20d2e96bd179c213 on integration/fx-team.
 0:15.87 Attempting to find a pushhead containing b0096c5c727749ad3e79cbdf20d2e96bd179c213 on integration/mozilla-inbound.
 0:15.87 Attempting to find a pushhead containing b0096c5c727749ad3e79cbdf20d2e96bd179c213 on releases/mozilla-aurora.
 0:16.72 Retrieving the last 50 pushheads starting with id 77464 on integration/mozilla-inbound
 0:16.72 Retrieving the last 50 pushheads starting with id 19399 on integration/fx-team
 0:16.72 Retrieving the last 50 pushheads starting with id 30289 on mozilla-central


and then it just sits there and if you wait a really really really long time it'll tell you it couldn't find any usable push heads.

This is very sad because full debug builds on my VM take a very long time.

I *think* we can fix by just updating artifacts.py or whatever it is to look for the new linux64-dbg artifacts. But the filenames also changed and seem to now all be "target.something", which might further upset it. I'm not sure yet. Chris, do you know why all the names changed?
Flags: needinfo?(cmanchester)
OK, so it's even worse because apparently we index into https://tools.taskcluster.net/index/artifacts/#buildbot.branches.mozilla-central/buildbot.branches.mozilla-central which as the name might suggest only has buildbot things.
I think we're not supposed to be using the buildbot index anymore anyway. At any rate, I'll take care of it.
Assignee: nobody → cmanchester
Flags: needinfo?(cmanchester)
According to mshal the buildbot route will include Taskcluster builds. It gives us the ability to query by revision first, which we're using, so I think we need to stick with this for now.
Mike, I'm going to push a patch your way that allows us to keep using the buildbot route. If there's a relatively easy way to get this in gecko.v2, let me know.
I'm fine with continuing to use buildbot.* for now. Even if you switched to gecko.v2, it would still use 'linux32-dbg', since that is the build_name and build_type specified in the task definitions. The routes come from here: https://dxr.mozilla.org/mozilla-central/source/testing/taskcluster/routes.json

A few of the task definitions specify 'dbg', which should be changed to 'debug':

./tasks/builds/dbg_linux32.yml:    build_type: 'dbg'
./tasks/builds/dbg_macosx64.yml:    build_type: 'dbg'
./tasks/builds/dbg_linux64.yml:    build_type: 'dbg'

Other tasks in taskcluster use 'debug', and mozharness (which is responsible for setting up the routes in buildbot builds) uses 'debug'.

Additionally, it seems the buildbot builds used 'linux' as the platform name for linux32, whereas taskcluster now uses 'linux32'. We should probably change these back to just 'linux' for consistency. It might be worth double-checking the others as well - here's the list of current platform names used in mozharness:

./releng_base_android_64_builds.py:    'platform': 'android',
./releng_base_mac_64_cross_builds.py:    'platform': 'macosx64',
./releng_base_windows_32_builds.py:    'platform': 'win32',
./releng_base_linux_32_builds.py:    'platform': 'linux',
./releng_base_windows_64_builds.py:    'platform': 'win64',
./releng_base_linux_64_builds.py:    'platform': 'linux64',
./releng_base_mac_64_builds.py:    'platform': 'macosx64',
./releng_sub_linux_configs/64_stat_and_opt.py:    'platform': 'linux64',
./releng_sub_linux_configs/64_artifact.py:    'platform': 'linux64',
Yes, let's please get these fixed, even if it causes some momentary pain.  I noticed the difference a while back, but wasn't sure which was the "right" way to name the platforms.  Ideally the right names would end up in taskcluster/docs/attributes.rst.
There's a related problem, which is that we appear to be sending both opt and debug linux64 builds to the same route right now. Before we turned off buildbot builds for this platform, maybe this was helping us a bit, because they wouldn't conflict with the buildbot debug builds (and I guess we're getting this effect from the naming inconsistency on other platforms).

For now I have a band-aid patch for linux.
Attachment #8757356 - Flags: review?(mshal) → review+
Comment on attachment 8757356 [details]
MozReview Request: Bug 1276037 - Update routes for linux64 debug builds to reflect that they are debug builds. r=mshal

https://reviewboard.mozilla.org/r/55648/#review52604

Should we do the same for macosx64 debug?
Attachment #8757357 - Flags: review?(mshal) → review+
Comment on attachment 8757357 [details]
MozReview Request: Bug 1276037 - Update artifact builds to be able to find Linux taskcluster builds. r=mshal

https://reviewboard.mozilla.org/r/55650/#review52614
Can we fix the gecko.v2 routes as well here? Or file a followup?
https://reviewboard.mozilla.org/r/55648/#review52604

I'm not sure, it seems like this might create confusion with the existing builds uploaded to that route by buildbot.
Blocks: 1276352
(In reply to Michael Shal [:mshal] from comment #13)
> Can we fix the gecko.v2 routes as well here? Or file a followup?

File bug 1276352. Thanks for the speedy reviews!
https://hg.mozilla.org/mozilla-central/rev/00e5d65e076c
https://hg.mozilla.org/mozilla-central/rev/e2261f30d0a7
Status: NEW → RESOLVED
Closed: 4 years ago
Resolution: --- → FIXED
Target Milestone: --- → Firefox 49
Component: Build Config → General
Product: Firefox → Firefox Build System
Target Milestone: Firefox 49 → mozilla49
You need to log in before you can comment on or make changes to this bug.