1372412 - OS X opt builds not being generated on mozillabeta

Status: RESOLVED FIXED

(Release Engineering :: General, defect)

Description

[Phil Ringnalda (:philor)]

They're kinda a big deal, we'd like to have them so we could hypothetically open the tree at some point.

Comment 1

[Phil Ringnalda (:philor)]

Possibly related, possibly completely separate, we also aren't running tests on the OS X DevEdition builds, though at least we are running those builds.

Comment 2

[Aki Sasaki (not active)]

The patch I'm about to submit enables these tc builds/tests:

>     "label": "build-macosx64/opt",
>     "label": "test-macosx64/opt-cppunit",
>     "label": "test-macosx64/opt-crashtest",
>     "label": "test-macosx64/opt-crashtest-e10s",
>     "label": "test-macosx64/opt-external-media-tests-base",
>     "label": "test-macosx64/opt-external-media-tests-youtube",
>     "label": "test-macosx64/opt-gtest",
>     "label": "test-macosx64/opt-jittest",
>     "label": "test-macosx64/opt-jsreftest-e10s-1",
>     "label": "test-macosx64/opt-jsreftest-e10s-2",
>     "label": "test-macosx64/opt-marionette-e10s",
>     "label": "test-macosx64/opt-mochitest-1",
>     "label": "test-macosx64/opt-mochitest-2",
>     "label": "test-macosx64/opt-mochitest-3",
>     "label": "test-macosx64/opt-mochitest-4",
>     "label": "test-macosx64/opt-mochitest-5",
>     "label": "test-macosx64/opt-mochitest-a11y",
>     "label": "test-macosx64/opt-mochitest-browser-chrome-1",
>     "label": "test-macosx64/opt-mochitest-browser-chrome-2",
>     "label": "test-macosx64/opt-mochitest-browser-chrome-3",
>     "label": "test-macosx64/opt-mochitest-browser-chrome-4",
>     "label": "test-macosx64/opt-mochitest-browser-chrome-5",
>     "label": "test-macosx64/opt-mochitest-browser-chrome-6",
>     "label": "test-macosx64/opt-mochitest-browser-chrome-7",
>     "label": "test-macosx64/opt-mochitest-browser-chrome-e10s-1",
>     "label": "test-macosx64/opt-mochitest-browser-chrome-e10s-2",
>     "label": "test-macosx64/opt-mochitest-browser-chrome-e10s-3",
>     "label": "test-macosx64/opt-mochitest-browser-chrome-e10s-4",
>     "label": "test-macosx64/opt-mochitest-browser-chrome-e10s-5",
>     "label": "test-macosx64/opt-mochitest-browser-chrome-e10s-6",
>     "label": "test-macosx64/opt-mochitest-browser-chrome-e10s-7",
>     "label": "test-macosx64/opt-mochitest-chrome-1",
>     "label": "test-macosx64/opt-mochitest-chrome-2",
>     "label": "test-macosx64/opt-mochitest-chrome-3",
>     "label": "test-macosx64/opt-mochitest-clipboard",
>     "label": "test-macosx64/opt-mochitest-clipboard-e10s",
>     "label": "test-macosx64/opt-mochitest-devtools-chrome-1",
>     "label": "test-macosx64/opt-mochitest-devtools-chrome-2",
>     "label": "test-macosx64/opt-mochitest-devtools-chrome-3",
>     "label": "test-macosx64/opt-mochitest-devtools-chrome-4",
>     "label": "test-macosx64/opt-mochitest-devtools-chrome-5",
>     "label": "test-macosx64/opt-mochitest-devtools-chrome-6",
>     "label": "test-macosx64/opt-mochitest-devtools-chrome-7",
>     "label": "test-macosx64/opt-mochitest-devtools-chrome-8",
>     "label": "test-macosx64/opt-mochitest-devtools-chrome-e10s-1",
>     "label": "test-macosx64/opt-mochitest-devtools-chrome-e10s-2",
>     "label": "test-macosx64/opt-mochitest-devtools-chrome-e10s-3",
>     "label": "test-macosx64/opt-mochitest-devtools-chrome-e10s-4",
>     "label": "test-macosx64/opt-mochitest-devtools-chrome-e10s-5",
>     "label": "test-macosx64/opt-mochitest-devtools-chrome-e10s-6",
>     "label": "test-macosx64/opt-mochitest-devtools-chrome-e10s-7",
>     "label": "test-macosx64/opt-mochitest-devtools-chrome-e10s-8",
>     "label": "test-macosx64/opt-mochitest-e10s-1",
>     "label": "test-macosx64/opt-mochitest-e10s-2",
>     "label": "test-macosx64/opt-mochitest-e10s-3",
>     "label": "test-macosx64/opt-mochitest-e10s-4",
>     "label": "test-macosx64/opt-mochitest-e10s-5",
>     "label": "test-macosx64/opt-mochitest-gpu",
>     "label": "test-macosx64/opt-mochitest-gpu-e10s",
>     "label": "test-macosx64/opt-mochitest-jetpack",
>     "label": "test-macosx64/opt-mochitest-media",
>     "label": "test-macosx64/opt-mochitest-media-e10s",
>     "label": "test-macosx64/opt-mochitest-webgl-e10s-1",
>     "label": "test-macosx64/opt-mochitest-webgl-e10s-2",
>     "label": "test-macosx64/opt-mochitest-webgl-e10s-3",
>     "label": "test-macosx64/opt-reftest-e10s",
>     "label": "test-macosx64/opt-talos-chrome-e10s",
>     "label": "test-macosx64/opt-talos-dromaeojs-e10s",
>     "label": "test-macosx64/opt-talos-g1-e10s",
>     "label": "test-macosx64/opt-talos-g2-e10s",
>     "label": "test-macosx64/opt-talos-g4-e10s",
>     "label": "test-macosx64/opt-talos-g5-e10s",
>     "label": "test-macosx64/opt-talos-other-e10s",
>     "label": "test-macosx64/opt-talos-svgr-e10s",
>     "label": "test-macosx64/opt-talos-tp5o-e10s",
>     "label": "test-macosx64/opt-web-platform-tests-e10s-1",
>     "label": "test-macosx64/opt-web-platform-tests-e10s-2",
>     "label": "test-macosx64/opt-web-platform-tests-e10s-3",
>     "label": "test-macosx64/opt-web-platform-tests-e10s-4",
>     "label": "test-macosx64/opt-web-platform-tests-e10s-5",
>     "label": "test-macosx64/opt-web-platform-tests-reftests-e10s",
>     "label": "test-macosx64/opt-xpcshell",

Comment 3

[Aki Sasaki (not active)]

Attachment: bug 1372412 - reenable osx opt builds on beta.

This patch moves the mozilla_beta_tasks filter() function out, so we can call it from mozilla_release_tasks.  This is mainly so we can disable osx opt tc builds on release, while enabling on beta.

Review commit: https://reviewboard.mozilla.org/r/148258/diff/#index_header
See other reviews: https://reviewboard.mozilla.org/r/148258/

Comment 4

[Aki Sasaki (not active)]

Attachment: bug 1372412 - reenable osx opt builds on beta.

Review commit: https://reviewboard.mozilla.org/r/148260/diff/#index_header
See other reviews: https://reviewboard.mozilla.org/r/148260/

Comment 5

[Aki Sasaki (not active)]

Comment on attachment 8876934 [details]
bug 1372412 - reenable osx opt builds on beta.

irc catlee r+. landed on beta.

Comment 6

[Aki Sasaki (not active)]

Aiui, devedition only runs on mozilla-beta.

In bug 1350413, we added macosx builds in-tree, scheduled through bbb.
In bug 1353376, we turned off all macosx buildbot-scheduled builds, for fx >= 55.

These two changes meant we were going to be missing devedition on beta as soon as 55 reached it, but we didn't see that until today because no other tree has devedition builds.

We need to either:

- hack on top of the hack: modify the bug 1353376 hack to allow devedition schedulers, or
- some sort of real fix, which may include backing out the above changes until 56.

Longer term, we may want to consider enabling devedition builds or nightlies on central, just to make sure they're working.

Comment 7

[Aki Sasaki (not active)]

(In reply to Aki Sasaki [:aki] from comment #6)
> Aiui, devedition only runs on mozilla-beta.
> 
> In bug 1350413, we added macosx builds in-tree, scheduled through bbb.
> In bug 1353376, we turned off all macosx buildbot-scheduled builds, for fx
> >= 55.
> 
> These two changes meant we were going to be missing devedition on beta as
> soon as 55 reached it, but we didn't see that until today because no other
> tree has devedition builds.

This may be inaccurate: devedition may have landed afterwards.


https://hg.mozilla.org/releases/mozilla-beta/rev/2d555a2ddb67
https://hg.mozilla.org/releases/mozilla-beta/rev/496fd4521a94

These are beta only; we need fixes on central. These will likely diverge yet again bc we're planning on moving osx to x-compiled builds. We need to remember devedition!

Comment 8

[Aki Sasaki (not active)]

It looks like the only change that broke osx devedition tests may have been the removal of the sendchange.

Comment 9

[Aki Sasaki (not active)]

https://hg.mozilla.org/integration/mozilla-inbound/rev/abdf1eb7663f1986f69ff336ede53e5550d73323
bug 1372412 - reenable osx opt builds on beta. r=catlee a=release CLOSED TREE

https://hg.mozilla.org/integration/mozilla-inbound/rev/fcc48fdcf68ad5d3b36c9fb9a200199586265d10
bug 1372412 - enable macosx devedition tests on beta. r=catlee a=release CLOSED TREE

Comment 10

[Aki Sasaki (not active)]

I believe we're good here.

Comment 11

[Joel Maher ( :jmaher ) (UTC -8)]

I am seeing a problem with this as we are using enable_talos_sendchange=True and this is causing q1 to fail as per bug 1372449 (we only run q1 on windows as it only works there- we have builders for linux/osx if we can get time to work on it).  In fact the only instances of enable_talos_sendchange=true is for windows, not any osx/linux.

:aki, can you fix this issue either here or in bug 1372449 ?

Comment 12

[Ryan VanderMeulen [:RyanVM]]

https://hg.mozilla.org/mozilla-central/rev/abdf1eb7663f
https://hg.mozilla.org/mozilla-central/rev/fcc48fdcf68a

Comment 13

[Aki Sasaki (not active)]

(In reply to Joel Maher ( :jmaher) from comment #11)
> I am seeing a problem with this as we are using enable_talos_sendchange=True
> and this is causing q1 to fail as per bug 1372449 (we only run q1 on windows
> as it only works there- we have builders for linux/osx if we can get time to
> work on it).  In fact the only instances of enable_talos_sendchange=true is
> for windows, not any osx/linux.
> 
> :aki, can you fix this issue either here or in bug 1372449 ?

Sure, patch + comment in that bug.